🔧 Fix Pipelines & automation Automation & Pipelines · step 24 of 26 Updated

Relative Paths Break When a GIS Script Runs Automatically: How to Fix It

Problem statement

Every path in the script is relative, which felt tidy. Then the same script runs from a scheduler, a different folder, or a colleague's machine:

DriverError: data/raw/parcels.shp: No such file or directory

The file is there. What changed is where the process was standing when it resolved the name. A relative path is not a location — it is a location plus an assumption about the current working directory, and that assumption is silently different in every context that is not your terminal.

Symptoms that all trace back to this:

  • FileNotFoundError for a path you can see in the message
  • outputs appear in your home directory instead of the project
  • a config file loads in the IDE and not from the command line
  • pytest passes and the scheduled run fails on the same code
  • a log file is created in one place today and another tomorrow
  • Path("~/data/out") resolves to a literal folder called ~

Common causes:

  • the working directory is set by the launcher: an IDE uses the project root, cron the user's home, systemd /
  • the script is invoked from a different folder than the one it lives in
  • a relative path is resolved at import time but used after a chdir
  • ~ is never expanded because it is a shell feature, not a filesystem one
  • paths are built with string concatenation and pick up the wrong separator on Windows

Quick answer

To make paths behave identically in every context:

  1. anchor everything to the script's own location: BASE = Path(__file__).resolve().parent
  2. build paths with the / operator on Path objects, never string concatenation
  3. call .expanduser() on anything a human might have written with ~
  4. resolve paths once, at start-up, and store them — do not recompute mid-run
  5. print the resolved paths on start-up so the log records exactly what was used
from pathlib import Path

BASE = Path(__file__).resolve().parent      # the folder holding this script
DATA = BASE / "data"
SRC  = DATA / "raw"
OUT  = DATA / "out"
OUT.mkdir(parents=True, exist_ok=True)

print("base:", BASE)
print("src :", SRC, "exists:", SRC.is_dir())

__file__ is the one anchor that does not depend on how the program was started. Everything else — the working directory, PATH, environment variables — is set by the launcher.

Where a relative path resolves

The same relative path resolved from three launch contexts to three different absolute paths.
One relative path, three launchers, three different files — none of them wrong, exactly.

Step-by-step solution

Grid comparing four path anchoring strategies across portability, config-friendliness and testability.
Four anchors, and when each is the right one.

See what the process actually thinks

Two lines at the top of the script remove all guesswork from the log.

from pathlib import Path
import sys

print("cwd       :", Path.cwd())
print("script    :", Path(__file__).resolve())
print("argv[0]   :", sys.argv[0])

If cwd is /home/gis under the scheduler and /srv/gis in your terminal, every relative path in the script points somewhere different in the two runs — and that is the whole bug.

Anchor to the script, not the caller 

from pathlib import Path

BASE = Path(__file__).resolve().parent

# for a script one level down, e.g. project/scripts/run.py
PROJECT_ROOT = Path(__file__).resolve().parents[1]

.resolve() matters: it turns a relative invocation (python scripts/run.py) into an absolute path and follows symlinks, so BASE is stable regardless of how the script was launched.

Prefer configuration over hard-coded structure

Anchoring to __file__ is right for code that ships with its data layout. When the data lives elsewhere — a network share, a mounted volume, a per-environment location — put the root in configuration and resolve it once.

import os
from pathlib import Path

BASE = Path(__file__).resolve().parent
DATA_ROOT = Path(os.environ.get("GIS_DATA_ROOT", BASE / "data")).expanduser().resolve()

SRC = DATA_ROOT / "raw"
OUT = DATA_ROOT / "out"

The default keeps development frictionless; the environment variable lets the scheduled deployment point at /mnt/gisdata without a code change.

Expand ~ yourself

~ is expanded by the shell. Python never does it implicitly, so a config value of ~/gisdata/out creates a directory literally named ~.

from pathlib import Path

p = Path("~/gisdata/out")
print(p.exists())                    # False — there is no folder called "~"

p = Path("~/gisdata/out").expanduser()
print(p)                             # /home/gis/gisdata/out

Make expanduser() part of the single place where you turn config strings into paths, so nothing downstream has to remember.

Never build paths by concatenating strings 

# fragile: wrong separator on Windows, doubles or drops slashes
path = out_dir + "/" + name + ".gpkg"

# correct
from pathlib import Path
path = Path(out_dir) / f"{name}.gpkg"

Path.__truediv__ inserts the right separator for the platform and normalises duplicates. It also composes with with_suffix(), with_stem() and relative_to(), which string paths do not.

Resolve once, at start-up

A path resolved lazily inside a loop can change meaning if anything calls os.chdir() — including some third-party libraries and notebook magics.

# fragile: meaning depends on the cwd at call time
def out_path(name):
    return Path("data/out") / name

# robust: resolved once, immune to later chdir
OUT = (Path(__file__).resolve().parent / "data" / "out").resolve()
def out_path(name):
    return OUT / name

A small Paths dataclass built once and passed around makes this explicit and testable:

from dataclasses import dataclass
from pathlib import Path

@dataclass(frozen=True)
class Paths:
    base: Path
    src: Path
    out: Path
    logs: Path

    @classmethod
    def from_root(cls, root: Path) -> "Paths":
        root = Path(root).expanduser().resolve()
        return cls(root, root / "data/raw", root / "data/out", root / "logs")

PATHS = Paths.from_root(Path(__file__).resolve().parent)

Fail early with a message that names the absolute path

Most of the pain in this class of bug is that the error names a relative path, which tells you nothing about where it looked.

import sys

for label, p in (("input", PATHS.src), ("output", PATHS.out.parent)):
    if not p.exists():
        sys.exit(f"{label} path does not exist: {p}  (cwd={Path.cwd()})")

Printing both the resolved path and the working directory turns a two-hour investigation into a five-second read.

Code examples

Example 1: a complete path setup block 

"""update_parcels.py — runs identically from a terminal, an IDE, or a scheduler."""
from pathlib import Path
import os, sys

BASE = Path(__file__).resolve().parent
DATA = Path(os.environ.get("GIS_DATA_ROOT", BASE / "data")).expanduser().resolve()

SRC  = DATA / "raw"
OUT  = DATA / "out"
LOGS = BASE / "logs"

for p in (OUT, LOGS):
    p.mkdir(parents=True, exist_ok=True)

if not SRC.is_dir():
    sys.exit(f"input folder missing: {SRC} (cwd={Path.cwd()})")

print(f"base={BASE}\ndata={DATA}\nsrc={SRC}\nout={OUT}", flush=True)

Example 2: resolving paths declared in a config file

Paths written in a YAML file should be interpreted relative to the config file itself — that is what a reader expects.

from pathlib import Path
import yaml

def load_config(config_path: str | Path) -> dict:
    config_path = Path(config_path).expanduser().resolve()
    cfg = yaml.safe_load(config_path.read_text(encoding="utf-8"))
    root = config_path.parent

    for key in ("input_dir", "output_dir", "boundary"):
        if key in cfg:
            cfg[key] = (root / Path(cfg[key]).expanduser()).resolve()
    return cfg

cfg = load_config("configs/daily.yml")
print(cfg["input_dir"])       # absolute, no matter where you ran it from

root / absolute_path returns the absolute path unchanged, so users can give either form and both work.

Example 3: a temporary working directory, safely

If a library insists on relative paths, change directory in a context manager so the change cannot leak.

import os
from contextlib import contextmanager
from pathlib import Path

@contextmanager
def working_dir(path: Path):
    previous = Path.cwd()
    os.chdir(path)
    try:
        yield Path(path)
    finally:
        os.chdir(previous)

with working_dir(PATHS.out):
    legacy_tool_that_writes_here()

Example 4: keeping test data next to the tests 

from pathlib import Path
import geopandas as gpd

FIXTURES = Path(__file__).resolve().parent / "fixtures"

def test_reads_sample_parcels():
    gdf = gpd.read_file(FIXTURES / "parcels_sample.gpkg")
    assert len(gdf) == 12

Tests are the context where working directories vary most — pytest from the root, an IDE from the test folder, CI from a checkout path. Anchoring to __file__ makes them all agree.

Explanation

Every process has a current working directory, inherited from whatever started it. A relative path is resolved against that directory at the moment the filesystem call is made. So Path("data/raw") does not name a folder; it names "a folder called data/raw beneath wherever this process happens to be".

Checklist of path hygiene rules for automated scripts.
Six habits that make paths behave the same everywhere.

Interactive work hides the assumption because your terminal and your IDE both tend to sit in the project root. Automation exposes it: cron uses the user's home directory, systemd uses / unless WorkingDirectory= is set, Task Scheduler uses whatever is in "Start in" — often C:\Windows\System32 — and a CI runner uses a checkout path that changes per build.

__file__ is different in kind. It is set by the import machinery from the path used to load the module, so Path(__file__).resolve() gives the file's real location on disk regardless of the launcher. That makes it the natural anchor for anything that ships alongside the code: templates, fixtures, small reference datasets, the default data folder.

It is not the right anchor for everything. Large or shared datasets do not belong next to the source, and their location genuinely differs between a laptop and a server. That is a configuration concern, so the pattern is a two-step one: anchor the code-relative things to __file__, take the data root from config or environment with a sensible default, resolve both to absolute paths once at start-up, and pass those objects around. After that, no function anywhere in the program needs to know what the working directory is — which is precisely the property that makes the script portable.

Edge cases or notes

  • __file__ in frozen apps and notebooks: PyInstaller sets it to the bundle location (use sys._MEIPASS when frozen), and Jupyter does not define it at all — use Path.cwd() or a config value in notebooks.
  • resolve() follows symlinks: On a deployment that symlinks current -> releases/2026-08-11, resolving turns a stable path into a versioned one. Use Path(__file__).parent.absolute() if you need the symlinked form.
  • Windows path length: Paths longer than 260 characters fail unless long-path support is enabled. Mirrored deep trees hit this surprisingly often.
  • UNC paths and mapped drives: A scheduled task running as SYSTEM cannot see Z:\. Always use \\server\share\... in automation.
  • os.chdir() is process-global: It affects threads and every library in the process. Prefer absolute paths over changing directory.
  • Trailing separators matter to some drivers: GDAL treats dir/ and dir alike for most drivers, but a few virtual filesystems do not. Path normalises this for you.
  • Case sensitivity differs by platform: Data/Raw works on Windows and fails on Linux. Keep directory names lower-case in automated trees.

FAQ

Why does my script find the data in the IDE but not from cron?

The IDE sets the working directory to the project root; cron sets it to the user's home. Relative paths therefore resolve to different places. Anchor to Path(__file__).resolve().parent and the launcher stops mattering.

Should I use os.chdir() at the start of the script?

It is a reasonable belt-and-braces measure, but it does not replace absolute paths — a library that changes directory later will break you again. Resolve paths once at start-up and use the resolved objects.

Why does Path("~/data") not work?

Tilde expansion is a shell feature. Python treats ~ as an ordinary directory name, so you must call .expanduser() on any path that might contain it — typically anything read from config or the command line.

How should paths in a config file be interpreted?

Relative to the config file, which is what a human reader assumes. Resolve them at load time with (config_path.parent / value).resolve(); absolute values pass through unchanged.

What is the difference between resolve() and absolute()?

Both make a path absolute, but resolve() also normalises .. segments and follows symlinks. Prefer resolve() except when a symlinked deployment path is deliberately part of your layout.

Does Path(__file__) work inside a Jupyter notebook?

No — notebooks do not define __file__. Use Path.cwd(), or better, keep pipeline code in a .py module that the notebook imports, so the module can anchor itself properly.

How do I make outputs land next to the input tree?

Compute the input's path relative to the search root with relative_to(), then join it onto the output root. That preserves the folder structure and prevents same-named files from overwriting each other.