🔧 Fix Pipelines & automation Automation & Pipelines · step 23 of 26 Updated

ModuleNotFoundError in a Scheduled GIS Job: Fixing the Wrong Python Environment

Problem statement

The script imports GeoPandas without complaint in your terminal. The scheduler's log says:

Traceback (most recent call last):
  File "/srv/gis/update_parcels.py", line 3, in 
    import geopandas as gpd
ModuleNotFoundError: No module named 'geopandas'

The package is installed — just not for the interpreter that ran the job. A ModuleNotFoundError in an automated context is almost never a broken install; it is a script executed by a different Python than the one you installed into.

Where the mismatch comes from:

  • cron, systemd or Task Scheduler resolved python on a minimal PATH to the system interpreter
  • the virtualenv was "activated" in a shell profile the scheduler never reads
  • a conda environment needs conda activate, which is a shell function, not a program
  • the shebang line says #!/usr/bin/python3 while the packages live in a venv
  • the job runs as a different user with a different home and a different install
  • the package was installed with pip install --user for your account only
  • the script imports a sibling module of its own that is not on sys.path when run from elsewhere

The same error text covers two distinct problems — the wrong interpreter, and the right interpreter with the wrong sys.path — and they have different fixes.

Quick answer

To fix a ModuleNotFoundError that only appears when scheduled:

  1. print sys.executable and sys.path from inside the failing job
  2. compare with the same two values from your working manual run
  3. invoke the venv's interpreter by absolute path — never python plus activation
  4. for conda, use conda run -n <env> or the env's bin/python directly
  5. for your own modules, install the project (pip install -e .) instead of relying on the working directory
# crontab: absolute interpreter, absolute script, captured output
30 2 * * * /srv/gis/.venv/bin/python /srv/gis/update_parcels.py >> /srv/gis/logs/job.log 2>&1
# top of the script, permanently — it costs nothing and answers the question instantly
import sys
print("interpreter:", sys.executable)
print("sys.path[0:3]:", sys.path[:3])

A virtualenv's bin/python configures sys.path for that environment on start-up. Activation exists only to put it first on your shell's PATH; it is not required to run anything.

Which Python ran the job?

Two interpreters — the venv one with packages and the system one without — and the same script run by both.
One script, two interpreters, two different `sys.path`s — only one of them has GeoPandas.

Step-by-step solution

Checklist for diagnosing which interpreter and path a scheduled job used.
Six checks that separate "wrong interpreter" from "right interpreter, wrong path".

Make the job report its own interpreter

Guessing is expensive when each test costs a scheduling cycle. Put the diagnostics in the script and read them from the log.

import sys, site, os
from pathlib import Path

print("executable :", sys.executable)
print("version    :", sys.version.split()[0])
print("prefix     :", sys.prefix)
print("base prefix:", sys.base_prefix)      # differs from prefix inside a venv
print("cwd        :", Path.cwd())
print("user       :", os.environ.get("USER") or os.environ.get("USERNAME"))
print("VIRTUAL_ENV:", os.environ.get("VIRTUAL_ENV"))
print("PYTHONPATH :", os.environ.get("PYTHONPATH"))
print("site-pkgs  :", site.getsitepackages())

sys.prefix != sys.base_prefix is the reliable test for "running inside a virtualenv". If they are equal under the scheduler and different in your terminal, the interpreter is the problem.

Find the interpreter that actually has the package 

# where is the package installed?
/srv/gis/.venv/bin/python -c "import geopandas, sys; print(sys.executable, geopandas.__file__)"

# what does the scheduler's bare `python` resolve to?
env -i /bin/sh -c 'command -v python python3; python3 -c "import sys; print(sys.executable)"'

Run both and compare. In nine cases out of ten the second prints /usr/bin/python3, which has no GIS stack installed.

Call the venv interpreter by absolute path

This is the fix, and it is a one-line change in the schedule.

# fragile — depends on PATH and a profile the scheduler never reads
30 2 * * * python /srv/gis/update_parcels.py

# robust — no activation needed
30 2 * * * /srv/gis/.venv/bin/python /srv/gis/update_parcels.py

On Windows Task Scheduler, the equivalent is C:\srv\gis\.venv\Scripts\python.exe as the program, with the script path as the argument and the project folder as "Start in".

Handle conda environments correctly

conda activate is a shell function defined by conda init; a scheduler running /bin/sh has never heard of it. Two supported options:

# 1. run through conda, which sets up the environment itself
30 2 * * * /opt/miniconda3/bin/conda run -n gis --no-capture-output python /srv/gis/update_parcels.py >> /srv/gis/logs/job.log 2>&1

# 2. call the environment's interpreter directly (fine for most GIS work)
30 2 * * * /opt/miniconda3/envs/gis/bin/python /srv/gis/update_parcels.py >> /srv/gis/logs/job.log 2>&1

Prefer conda run when the packages rely on activation scripts that set PROJ_LIB, GDAL_DATA or PROJ_NETWORK — GDAL builds from conda-forge usually do. Without those variables you trade ModuleNotFoundError for a PROJ database error, which is the same class of problem one layer down.

Fix the shebang, or stop relying on it

If the schedule invokes the script directly (/srv/gis/update_parcels.py), the shebang decides the interpreter.

#!/srv/gis/.venv/bin/python
chmod +x /srv/gis/update_parcels.py

#!/usr/bin/env python3 is portable but resolves through PATH — which is exactly the variable the scheduler does not set the way you expect. Point the shebang at the venv, or invoke the interpreter explicitly and let the shebang be irrelevant.

Check that the job runs as the right user

pip install --user installs into ~/.local/lib/python3.x/site-packages. A job that runs as gis, root, or SYSTEM has a different home directory and therefore cannot see it.

# what the job sees
30 2 * * * id -un && /usr/bin/python3 -c "import site; print(site.getusersitepackages())"

The fix is to install into a shared virtualenv owned by the service account rather than into a personal user site directory.

Fix imports of your own modules

A different error shape — No module named 'mypipeline' for a package that is your own code — usually means sys.path lacks the project root. Python adds the script's directory to sys.path, not the current working directory, so a script in scripts/ cannot import a package in the parent folder.

project/
  mypipeline/__init__.py
  scripts/run_daily.py       # `import mypipeline` fails here

Three fixes, best first:

# 1. install the project into the venv (editable), so it is importable from anywhere
/srv/gis/.venv/bin/pip install -e /srv/gis

# 2. run as a module from the project root
cd /srv/gis && /srv/gis/.venv/bin/python -m scripts.run_daily

# 3. last resort — set PYTHONPATH in the schedule
30 2 * * * PYTHONPATH=/srv/gis /srv/gis/.venv/bin/python /srv/gis/scripts/run_daily.py

Avoid sys.path.insert(0, "..") inside the script: it depends on the working directory and breaks the moment the file moves.

Code examples

Example 1: an import guard that explains itself 

import sys

REQUIRED = ("geopandas", "pyogrio", "shapely", "yaml")

missing = []
for name in REQUIRED:
    try:
        __import__(name)
    except ImportError:
        missing.append(name)

if missing:
    print(
        f"missing packages {missing}\n"
        f"  interpreter : {sys.executable}\n"
        f"  prefix      : {sys.prefix}\n"
        f"  in a venv   : {sys.prefix != sys.base_prefix}\n"
        f"  fix         : {sys.executable} -m pip install {' '.join(missing)}",
        file=sys.stderr,
    )
    raise SystemExit(2)

Printing sys.executable next to the pip command means whoever reads the log at 08:00 gets the fix, not a puzzle.

Example 2: pin the environment inside the wrapper 

#!/usr/bin/env bash
# /srv/gis/run.sh
set -euo pipefail

VENV="/srv/gis/.venv"
PY="$VENV/bin/python"

[ -x "$PY" ] || { echo "interpreter missing: $PY" >&2; exit 3; }
"$PY" -c "import geopandas" 2>/dev/null || { echo "geopandas not in $VENV" >&2; exit 3; }

cd /srv/gis
exec "$PY" update_parcels.py "$@"

The wrapper fails with a specific exit code and message before the job starts, which is far easier to alert on than a traceback.

Example 3: verify the environment in CI so it never drifts 

# tests/test_environment.py
import sys
import importlib

def test_running_in_project_venv():
    assert sys.prefix != sys.base_prefix, "not running inside a virtualenv"

def test_gis_stack_importable():
    for name in ("geopandas", "shapely", "pyogrio", "rasterio"):
        importlib.import_module(name)

def test_project_package_installed():
    import mypipeline                      # fails if `pip install -e .` was skipped
    assert mypipeline.__file__

Example 4: record the environment with every run 

import json, sys, subprocess
from pathlib import Path
from datetime import datetime, timezone

def write_env_manifest(dest: Path) -> None:
    freeze = subprocess.run(
        [sys.executable, "-m", "pip", "freeze"], capture_output=True, text=True, check=True
    ).stdout.splitlines()
    dest.write_text(json.dumps({
        "run_at": datetime.now(timezone.utc).isoformat(),
        "executable": sys.executable,
        "python": sys.version.split()[0],
        "packages": freeze,
    }, indent=2), encoding="utf-8")

write_env_manifest(Path("logs/env-manifest.json"))

When a job that worked for a year suddenly cannot import something, the manifest from the last good run tells you exactly what changed.

Explanation

Python resolves imports by searching sys.path, and sys.path is built at interpreter start-up from the interpreter's own location. A virtualenv works by placing a python binary in .venv/bin next to a pyvenv.cfg; when that binary starts, it sets sys.prefix to the venv and adds the venv's site-packages to the search path. Nothing else is involved — no activation, no environment variable, no magic.

Triage table mapping scheduled-job import failures to their fixes.
Six causes of a scheduled `ModuleNotFoundError`, each with its own correct fix.

That is why .venv/bin/python script.py is completely sufficient, and why activate is not. Activation prepends the venv's bin directory to PATH and sets VIRTUAL_ENV so that a bare python in that shell finds the right binary. A scheduler starts a fresh, non-login shell with a minimal PATH, so the bare name resolves to /usr/bin/python3 — a perfectly working interpreter that has never seen your packages.

Conda adds one wrinkle. Its environments are also just directories with a bin/python, so calling that binary directly usually works. But conda's GDAL and PROJ builds ship activation hooks that export PROJ_LIB, GDAL_DATA and friends, and skipping activation skips those. conda run -n env performs the activation programmatically, which is why it is the safer form in a scheduler.

The second family of failures — your own package not importing — comes from a different rule: Python puts the directory of the script being run at the front of sys.path, not the current working directory. Running scripts/run_daily.py therefore makes scripts/ importable and the project root not. Installing the project into the environment removes the ambiguity permanently, which is why pip install -e . is worth the two minutes it takes to add a pyproject.toml.

Edge cases or notes

  • python may not exist at all: Many distributions ship only python3. A crontab calling python then fails with "command not found", which cron reports by mail — that is, invisibly.
  • pip install without the venv: Running pip install geopandas with the system pip installs for the system Python. Use /path/to/.venv/bin/python -m pip install … so the interpreter and the installer always match.
  • Two packages, one import name: import osgeo comes from GDAL, import yaml from PyYAML. Install the distribution name, import the module name.
  • Windows service accounts have no drive mappings: A job running as SYSTEM cannot see Z:\. Use UNC paths.
  • Docker containers hit this too: If the image's ENTRYPOINT uses the system Python but the packages were installed into a venv in the image, the same mismatch appears. Set ENV PATH="/opt/venv/bin:$PATH".
  • PYTHONPATH is a blunt instrument: It applies to every Python started in that environment and can shadow packages unexpectedly. Prefer installing the project.
  • PyQGIS is not pip-installable: import qgis requires the interpreter that ships with QGIS, and needs PYTHONPATH pointing at the QGIS Python directory — a genuinely different problem with its own fix.

FAQ

Why does import geopandas work in my terminal but not in the scheduled job?

The scheduled job ran a different interpreter — usually /usr/bin/python3 instead of your virtualenv's Python. Print sys.executable in both contexts to confirm, then invoke the venv interpreter by absolute path.

Do I need to activate the virtualenv in cron?

No. Activation only edits PATH for an interactive shell. Running /path/to/.venv/bin/python script.py gives you the environment's packages without any activation step.

How do I schedule a conda environment?

Use /opt/miniconda3/bin/conda run -n myenv --no-capture-output python script.py. It performs activation programmatically, which matters because conda's GDAL and PROJ builds set library paths during activation.

Why can Python not find my own module?

Because sys.path gets the directory of the script being executed, not the project root. Install the project with pip install -e ., or run it as a module with python -m package.script from the project root.

What is the difference between sys.prefix and sys.base_prefix?

sys.prefix is the environment currently in use and sys.base_prefix is the interpreter it was created from. When they differ, you are inside a virtualenv — the cleanest programmatic check available.

Should I set PYTHONPATH to fix imports?

Only as a stopgap. It applies to every Python process in that environment and can shadow installed packages in confusing ways. Installing the project into the venv is more predictable and survives moves.

How do I keep this from happening again?

Pin the interpreter in a wrapper script, add an import preflight that fails with a clear message, and record pip freeze with each run. All three turn a mysterious overnight failure into a one-line log entry.