Python glob Is Not Finding All My Shapefiles: How to Fix It #

Problem statement #

The folder plainly contains 60 shapefiles. The batch script reports:

found 0 files in data/raw

Or worse, it finds 43 of them and quietly processes those, leaving 17 datasets missing from the output with no error at all. glob() does not raise when a pattern matches nothing — an empty match is a perfectly legal result — so a wrong pattern looks exactly like an empty folder.

Typical symptoms:

• glob("*.shp") returns nothing, but ls shows the files
• files ending in .SHP are skipped on Linux but found on Windows
• files inside subfolders are ignored
• Path.glob() returns generator objects that appear empty on a second pass
• the script works from your editor and finds nothing when run from cron

Common causes:

• the pattern is relative to the current working directory, not the script
• extension case does not match (.SHP, .Shp)
• the files are one level deeper and the pattern is not recursive
• a hidden or trailing space in the folder name
• the generator was already consumed by an earlier len() or loop
• special characters in the folder path ([, ], ?, *) are being treated as pattern syntax

Quick answer #

When a glob finds nothing or too little:

1. print the absolute path you are actually globbing — Path(pattern).resolve()
2. list the folder without a pattern first: list(Path(src).iterdir())[:10]
3. make the search case-insensitive by filtering on path.suffix.lower()
4. use rglob() (or glob("**/*.shp", recursive=True)) for subfolders
5. materialise the result with sorted(...) so it is a list, not a one-shot generator

from pathlib import Path

src = Path("data/raw").resolve()
print("looking in:", src, "exists:", src.is_dir())

files = sorted(p for p in src.rglob("*") if p.suffix.lower() == ".shp")
print(f"found {len(files)} shapefiles")
for p in files[:5]:
    print("  ", p.relative_to(src))

Filtering on suffix.lower() sidesteps the case problem entirely, and rglob walks subfolders. Sorting turns the generator into a stable, reusable list.

Why a glob misses files #

Step-by-step solution #

Prove which directory you are searching #

A relative pattern is resolved against the process working directory, which is the folder you launched Python from — not the folder the script lives in. That is why the same script behaves differently from an IDE, a terminal, and a scheduler.

from pathlib import Path
import os

print("cwd        :", Path.cwd())
print("script dir :", Path(__file__).resolve().parent)
print("target     :", Path("data/raw").resolve())
print("exists     :", Path("data/raw").is_dir())

If exists is False, the pattern was never going to match. Anchor the path to the script or to a configured root instead:

BASE = Path(__file__).resolve().parent
src = BASE / "data" / "raw"

List the folder before filtering it #

Before debugging the pattern, confirm what is actually there. iterdir() applies no pattern at all.

src = Path("data/raw")
for p in sorted(src.iterdir())[:20]:
    print(f"{'d' if p.is_dir() else 'f'}  {p.name!r}")

Printing with !r shows quotes, which reveals trailing spaces and invisible characters in names — a surprisingly common cause of "the file is right there".

Handle extension case explicitly #

Path.glob() is case-sensitive on Linux and macOS with case-sensitive volumes, and case-insensitive on Windows. A dataset exported by another tool may well be PARCELS.SHP.

# fragile: misses .SHP on Linux
files = list(src.glob("*.shp"))

# portable: one pass, case-insensitive
files = sorted(p for p in src.iterdir() if p.suffix.lower() == ".shp")

# or a character-class pattern, if you prefer to stay in glob syntax
files = sorted(src.glob("*.[sS][hH][pP]"))

The suffix.lower() filter is the one to prefer — it reads clearly and extends to a set of extensions.

Search subfolders deliberately #

glob("*.shp") matches only the immediate directory. Data delivered by region or year usually sits one or more levels down.

# immediate directory only
list(src.glob("*.shp"))

# every level below, pathlib
list(src.rglob("*.shp"))

# every level below, glob module — the ** needs recursive=True
import glob
glob.glob("data/raw/**/*.shp", recursive=True)

Without recursive=True, ** behaves like a single * in the glob module, which is the classic reason a "recursive" pattern quietly matches one level.

Do not consume the generator twice #

Path.glob() returns a generator. Once iterated, it is exhausted — so a len() or a preview loop leaves nothing for the real loop.

files = src.glob("*.shp")
print(len(list(files)))     # consumes it
for f in files:             # runs zero times
    ...

files = sorted(src.glob("*.shp"))   # a list: reusable, and in a stable order

Sorting also makes runs reproducible, which matters when you are trying to work out which file the batch stopped on.

Match multiple extensions in one pass #

GIS folders mix formats. Collect the extensions you accept into a set and filter once.

VECTOR_EXT = {".shp", ".gpkg", ".geojson", ".json", ".gml", ".kml", ".fgb"}

files = sorted(
    p for p in src.rglob("*")
    if p.is_file() and p.suffix.lower() in VECTOR_EXT
)

This is much easier to reason about than chaining several globs, and it keeps a single ordered list.

Escape special characters in the folder name #

Square brackets and question marks are glob syntax. A real folder called data/raw [2024] will not match a pattern built by string concatenation.

import glob

folder = "data/raw [2024]"
glob.glob(f"{folder}/*.shp")                       # matches nothing
glob.glob(f"{glob.escape(folder)}/*.shp")          # correct

# pathlib avoids the problem: only the pattern argument is glob syntax
from pathlib import Path
list(Path("data/raw [2024]").glob("*.shp"))        # fine

Code examples #

Example 1: a reusable, defensive file finder #

from pathlib import Path

def find_files(root, extensions, recursive=True):
    """Return a sorted list of files under `root` matching `extensions`."""
    root = Path(root).expanduser().resolve()
    if not root.is_dir():
        raise NotADirectoryError(f"input folder not found: {root}")

    wanted = {e.lower() if e.startswith(".") else f".{e.lower()}" for e in extensions}
    it = root.rglob("*") if recursive else root.iterdir()
    files = sorted(p for p in it if p.is_file() and p.suffix.lower() in wanted)

    if not files:
        sample = [p.name for p in sorted(root.iterdir())[:10]]
        raise FileNotFoundError(
            f"no {sorted(wanted)} files under {root}. Folder contains: {sample}"
        )
    return files

files = find_files("data/raw", [".shp", ".gpkg"])
print(f"{len(files)} files")

Raising with a sample of what is in the folder turns a silent zero-match into a message that explains itself.

Example 2: only shapefiles that have their sidecars #

A .shp without its .shx and .dbf will fail at read time. Filtering early keeps the failure list short.

from pathlib import Path

REQUIRED = (".shx", ".dbf")

complete, incomplete = [], []
for shp in sorted(Path("data/raw").rglob("*.shp")):
    missing = [ext for ext in REQUIRED if not shp.with_suffix(ext).exists()]
    (incomplete if missing else complete).append((shp, missing))

print(f"{len(complete)} complete, {len(incomplete)} incomplete")
for shp, missing in incomplete:
    print(f"  ! {shp.name} missing {missing}")

Example 3: skip temporary, hidden and lock files #

from pathlib import Path

def is_real_input(p: Path) -> bool:
    if p.name.startswith((".", "~", "$")):        # hidden / office lock files
        return False
    if p.suffix.lower() in {".lock", ".tmp", ".part"}:
        return False
    if any(part in {"__pycache__", ".git", ".ipynb_checkpoints"} for part in p.parts):
        return False
    return p.is_file()

files = sorted(p for p in Path("data/raw").rglob("*.shp") if is_real_input(p))

Cloud-sync folders in particular are full of partial files; a placeholder that is still downloading looks like a valid path and fails on read.

Example 4: mirror the input tree in the output #

from pathlib import Path

SRC = Path("data/raw").resolve()
OUT = Path("data/out").resolve()

for shp in sorted(SRC.rglob("*.shp")):
    rel = shp.relative_to(SRC).with_suffix(".gpkg")
    dest = OUT / rel
    dest.parent.mkdir(parents=True, exist_ok=True)
    print(f"{shp.relative_to(SRC)} → {dest.relative_to(OUT)}")

relative_to() is what makes a recursive search usable: it preserves the folder structure instead of flattening 200 files into one directory, where same-named outputs would overwrite each other.

Explanation #

Glob patterns are a matching language, not a search engine. Each pattern is evaluated against directory entries at a specific starting point, so three separate things must line up: the starting directory must be the one you think it is, the depth must be covered by the pattern, and the literal characters must match the filenames byte for byte, including case.

The starting directory is where most of the confusion lives. A relative pattern is resolved against the current working directory of the process, not the location of the source file. Editors usually set the working directory to the project root, terminals to wherever you happen to be, and schedulers to something arbitrary like / or the user's home. The same script, unchanged, therefore finds files in one context and nothing in another.

Case sensitivity is the second trap, and it is platform-dependent by design: the filesystem, not Python, decides whether PARCELS.SHP matches *.shp. Code that works on a Windows laptop and finds nothing on a Linux server is almost always hitting this. Comparing suffix.lower() against a set moves the decision into your code, where it behaves identically everywhere.

Finally, glob() returning an empty result is not an error condition in Python's view, so nothing is raised and nothing is logged. That silence is the real problem: a batch that finds zero files exits successfully, a scheduler records a green run, and the missing data is discovered weeks later. Validating the match count immediately after discovery — and failing loudly on zero — is the single most valuable line in the script.

Edge cases or notes #

• Hidden files are skipped by *: Both glob and pathlib treat a leading dot as hidden and will not match it with *. Use iterdir() if you genuinely need dotfiles.
• Symlinked folders: rglob follows symlinked directories, so a loop of symlinks can recurse for a long time. Guard with os.walk(followlinks=False) if the tree is untrusted.
• Path.glob("**") differs from rglob("*") subtly: rglob(p) is exactly glob("**/" + p). Mixing the two forms in one codebase invites confusion; pick rglob.
• Network shares can be slow: A recursive walk of a large SMB mount is expensive. Cache the file list to a manifest and reuse it rather than re-walking each run.
• .gdb and .gpkg are containers: A File Geodatabase is a directory and a GeoPackage can hold many layers. Enumerate layers with pyogrio.list_layers() or fiona.listlayers() rather than counting files.
• Zipped shapefiles do not appear at all: Read them with the zip:// virtual filesystem prefix, or unpack first — *.shp will never match inside an archive.

Internal links #

• How to Batch Process a Folder of GIS Files in Python: The Complete Workflow
• How to Process GIS Files in Nested Subfolders and Mirror the Output Tree
• How to Batch Process Multiple Shapefiles in Python
• GeoPandas Not Reading Shapefile: Common Causes and Fixes
• Batch Script Stops at the First Bad File: How to Keep It Running
• Relative Paths Break When a GIS Script Runs Automatically: How to Fix It

FAQ #

Why does glob find files in my terminal but not from a scheduler? #

Because the pattern is relative and the two contexts have different working directories. Anchor paths to Path(__file__).resolve().parent or to a configured root, and the behaviour becomes identical everywhere.

How do I make a glob case-insensitive? #

Filter in Python rather than in the pattern: p.suffix.lower() == ".shp". The character-class form *.[sS][hH][pP] also works but becomes unreadable for longer extensions.

Why is my ** pattern not recursing? #

With the glob module, ** only spans directories when you pass recursive=True. With pathlib, use rglob("*.shp") or glob("**/*.shp"), which are recursive by definition.

Why did my file list turn out empty on the second loop? #

Path.glob() returns a generator that can be iterated once. Wrap it in sorted() or list() as soon as you create it, which also gives you a stable order for reproducible runs.

How can I tell whether zero matches means "empty folder" or "wrong pattern"? #

List the directory with iterdir() and include a sample of the names in the error message. Seeing the actual filenames immediately distinguishes a missing folder, a wrong extension, and a genuinely empty directory.

Should I use os.walk, glob, or pathlib? #

pathlib for almost everything — it is the most readable and handles paths as objects. Reach for os.walk when you need to prune whole subtrees during the walk, or to avoid following symlinks.

How do I find shapefiles inside a zip archive? #

Use GDAL's virtual filesystem: gpd.read_file("zip://data/raw/parcels.zip!parcels.shp"), or list the archive with zipfile.ZipFile(...).namelist() first. A filesystem glob cannot see inside an archive.