Manage Directories with shutil and glob

You’ve built Python scripts that create files, write data, and maybe even clean up after themselves. But as soon as you need to organize dozens of reports, move backups into dated folders, or delete stale temp directories, looping over os.listdir() and building manual paths gets brittle, verbose, and error-prone. The real pain is not reading a single file—it’s managing entire directory trees at scale. That’s where Python’s shutil and glob modules step in, giving you high-level operations (copy, move, delete a whole folder) and pattern-based file discovery, all without reinventing the wheel.

The problem this lesson solves

When your project outgrows a handful of files, manual directory management becomes a nightmare. Imagine you have a data/ folder that accumulates weekly exports. You need to:

• Archive last week’s folder into a backups/ directory.
• Delete any folder whose name starts with tmp_.
• Find all .csv files nested three levels deep.

Using only os.path and shutil.rmtree is possible, but cobbling together recursion, pattern matching, and error handling for every script is tedious and fragile. The core problem is that Python’s low-level os module doesn’t offer modern, ergonomic tools for batch directory operations or file pattern matching. Without shutil and glob, you’d write more code, introduce more bugs, and spend time debugging path logic instead of solving the actual problem.

Core concept / mental model

Think of shutil as your file & folder assistant — it knows how to copy, move, rename, and delete entire directories in one call. glob is your pattern-finding scout — it searches for files and directories whose names match a wildcard pattern (*.csv, **/*.json), like a simplified regular expression for file paths.

Pro tip: shutil handles the heavy lifting of recursive operations, whereas glob replaces many manual os.listdir + fnmatch loops. Together they form the essential toolkit for file wrangling in Python.

Key terms

• shutil — Short for “shell utilities.” Functions like shutil.copy(), shutil.copytree(), shutil.move(), shutil.rmtree(), and shutil.make_archive().
• glob — Pattern matching using *, ?, and [], plus ** for recursive matching (Python 3.5+).
• Working directory — The folder your script considers “home”; affected by os.chdir() or relative paths.

How it works step by step

1. Import the modules

import shutil
import glob

Both are part of Python’s standard library — no pip install needed.

2. Working with shutil for directories

The table below summarizes the most common shutil directory functions:

Warning: shutil.rmtree() deletes without moving to a trash bin. Always test on non‑production data first.

3. Using glob to find directories (and files)

glob.glob(pathname, recursive=False) returns a list of path strings matching the given pattern. Enable recursive searching with recursive=True and **.

Hands-on walkthrough

Let’s build a practical example: we’ll create a simulated project structure, then use shutil and glob to manage it.

Setup: create test directories and files

import os
import shutil

# Create a demo workspace
os.makedirs('my_project/data/raw', exist_ok=True)
os.makedirs('my_project/data/processed', exist_ok=True)
os.makedirs('my_project/backups', exist_ok=True)
os.makedirs('my_project/temp/test', exist_ok=True)

# Create some dummy files
for folder in ['data/raw', 'data/processed', 'temp/test']:
    for i in range(3):
        open(f'my_project/{folder}/file_{i}.txt', 'w').close()

print("Workspace created.")

Expected output: No errors; the directory tree is ready.

Copy an entire directory tree

import shutil

# Copy the 'data' folder into 'backups'
shutil.copytree('my_project/data', 'my_project/backups/data_backup')
print("Data backed up.")

Check: backups/data_backup now contains raw/ and processed/ with all files.

Delete all temp directories using glob + shutil

import glob
import shutil

for temp_dir in glob.glob('my_project/**/temp*', recursive=True):
    if os.path.isdir(temp_dir):
        shutil.rmtree(temp_dir)
        print(f"Removed {temp_dir}")

Expected output:

Removed my_project/temp

Move one directory into another

shutil.move('my_project/data/processed', 'my_project/backups/processed_backup')
print("Moved processed folder.")

Now backups/ contains both data_backup and processed_backup.

Compare options / when to choose what

Rule of thumb: If it involves a whole directory tree (copy, move, delete) or pattern matching over many paths, reach for shutil and glob. If you’re handling a single file or need fine‑grained control (like filtering by file size or modification date), os.path and os.walk might still be better.

Troubleshooting & edge cases

“Destination already exists” errors with shutil.copytree

shutil.copytreeraisesFileExistsError` if the target folder already exists. Workaround:

if not os.path.exists('backups/new_folder'):
    shutil.copytree('source', 'backups/new_folder')
else:
    print("Target already exists. Use `shutil.rmtree()` first or rename.")

Symlink handling

shutil.copytree by default copies the content of symlinks (not the symlinks themselves). Use symlinks=True to preserve them:

shutil.copytree('src', 'dst', symlinks=True)

glob with hidden files

Files starting with . are not matched by default. Use glob.glob('.*') or glob.glob('**/.*', recursive=True) to include them.

Large directory deletion is slow

shutil.rmtree deletes each file and folder one call at a time. For huge trees with millions of files, consider using os.system('rm -rf dir') (Linux/macOS) or third‑party libraries like pyfilesystem2 for performance. But for most projects, shutil is perfectly adequate.

Path is a file, not a directory

shutil.copytree, shutil.rmtree, and shutil.move on a file (not a directory) will raise NotADirectoryError. Always check with os.path.isdir().

if os.path.isdir('my_target'):
    shutil.rmtree('my_target')
else:
    os.remove('my_target')

What you learned & what's next

You now know how to manage directories with shutil and glob.

• shutil gives you high‑level commands to copy, move, rename, and delete whole directory trees — no recursion boilerplate.
• glob lets you find files and directories by pattern, including recursive traversal with **.
• You can combine them: use glob to locate directories, then shutil.rmtree() or shutil.move() to act on them.

You’ve met the lesson’s objectives:

• ✅ Explain the core idea behind managing directories with shutil and glob.
• ✅ Complete a practical exercise — creating, copying, moving, and deleting folders.

Next up: You’ll learn how to Work with zip and tar archives — compressing and extracting bundles of files, a natural follow‑up to organizing directories. This will complete your file‑management toolkit.