How to Handle Credentials and Secrets in an Automated GIS Job

Problem statement

The pipeline needs a PostGIS password, an S3 key pair and an API token. The quickest way to make it work is the one everybody regrets:

engine = create_engine("postgresql://gis_user:[email protected]:5432/gis")

Committed on Tuesday, pushed to a shared repository, copied into a Jupyter notebook, pasted into a Slack thread, then found by a scanner eighteen months later — long after that password was reused on three other systems. Even without a leak, hard-coded credentials mean the job cannot move between environments, and rotating the password means editing code.

The problems compound in an automated context:

a credential in the source tree is in every clone and every git object, forever
a scheduled job cannot answer an interactive password prompt
a connection string leaks into logs, tracebacks, run records and error emails
.env files get committed by accident, or copied into a Docker image layer
a token pasted into a notebook ends up in the .ipynb output cells

Quick answer

Keep secrets out of code, out of logs, and out of images:

read credentials from environment variables, or from files whose path is in an environment variable
keep a .env for local development, and put it in .gitignore on day one
never log or print a connection string — build the URL only where it is used
in Docker and CI, use the platform's secret mechanism, not ENV or a COPY
fail with a clear message when a required secret is missing

import os
from sqlalchemy import create_engine
from sqlalchemy.engine import URL

def require_env(name: str) -> str:
    value = os.environ.get(name)
    if not value:
        raise RuntimeError(
            f"missing required environment variable {name}. "
            f"Set it in the environment, or in a .env file for local development."
        )
    return value

def make_engine():
    url = URL.create(
        "postgresql+psycopg",
        username=require_env("PGUSER"),
        password=require_env("PGPASSWORD"),      # never interpolated into a string
        host=os.environ.get("PGHOST", "localhost"),
        port=int(os.environ.get("PGPORT", 5432)),
        database=require_env("PGDATABASE"),
    )
    return create_engine(url)

engine = make_engine()
print("connected to", engine.url.render_as_string(hide_password=True))

URL.create handles escaping for you — a password containing @, / or # breaks a hand-built URL — and render_as_string(hide_password=True) is the only form that should ever reach a log.

Where secrets can live

Grid comparing hard-coded, .env file, environment variables, secret files and a secret manager. — Five places a credential can live, ranked by how badly each one fails.

Step-by-step solution

Triage table of ways secrets leak and the control that stops each one. — Seven leak paths — code is only the first, and rarely the one that gets you.

Get secrets out of the code

The rule is simple: the repository contains the names of secrets, never their values.

# config.py — names and defaults only
REQUIRED_SECRETS = ("PGUSER", "PGPASSWORD", "PGDATABASE")
OPTIONAL_SETTINGS = {"PGHOST": "localhost", "PGPORT": "5432", "GIS_DATA_ROOT": "./data"}

# .env.example — committed, documents what is needed, contains nothing real
PGUSER=gis_user
PGPASSWORD=changeme
PGDATABASE=gis
PGHOST=db.internal
AWS_ACCESS_KEY_ID=
AWS_SECRET_ACCESS_KEY=

# .gitignore — add these before you create the real file
.env
.env.*
!.env.example
secrets/
*.pem
*.key

Load a `.env` for local development only

from pathlib import Path
import os

def load_dotenv(path: Path = Path(".env"), override: bool = False) -> int:
    """Minimal .env loader — real environment variables win by default."""
    if not path.exists():
        return 0
    loaded = 0
    for line in path.read_text(encoding="utf-8").splitlines():
        line = line.strip()
        if not line or line.startswith("#") or "=" not in line:
            continue
        key, value = line.split("=", 1)
        key, value = key.strip(), value.strip().strip('"').strip("'")
        if override or key not in os.environ:
            os.environ[key] = value
            loaded += 1
    return loaded

load_dotenv()

Or use python-dotenv, which does the same with more edge cases handled:

from dotenv import load_dotenv
load_dotenv()                       # local only; in production the env is already set

Real environment variables must take precedence over the file, so a server's configuration is never overridden by a stray .env that got copied along with the code.

Prefer secret files over environment variables where you can

Environment variables are visible in /proc/<pid>/environ, in docker inspect, and in any subprocess. A file with 0600 permissions is tighter, and it is the mechanism Docker and Kubernetes secrets use.

import os
from pathlib import Path

def read_secret(name: str, default=None) -> str:
    """Read NAME_FILE if set (Docker/K8s style), else NAME, else default."""
    file_var = os.environ.get(f"{name}_FILE")
    if file_var:
        path = Path(file_var)
        if not path.exists():
            raise RuntimeError(f"{name}_FILE points at {path}, which does not exist")
        return path.read_text(encoding="utf-8").strip()
    value = os.environ.get(name, default)
    if value is None:
        raise RuntimeError(f"missing secret: set {name} or {name}_FILE")
    return value

password = read_secret("PGPASSWORD")

This one function supports plain env vars, Docker secrets (/run/secrets/...), Kubernetes mounted secrets and systemd credentials without any change to the call site.

Keep secrets out of logs and tracebacks

This is where most real leaks happen — not from the repository, but from an error email.

import logging, re

SECRET_PATTERN = re.compile(
    r"(password|passwd|pwd|token|secret|api[_-]?key|authorization)"
    r"(\s*[=:]\s*|%3D)([^\s&'\"]+)", re.IGNORECASE)

class RedactingFilter(logging.Filter):
    def filter(self, record: logging.LogRecord) -> bool:
        if isinstance(record.msg, str):
            record.msg = SECRET_PATTERN.sub(r"\1=***", record.msg)
            record.msg = re.sub(r"(://[^:/@\s]+):([^@\s]+)@", r"\1:***@", record.msg)
        return True

handler = logging.StreamHandler()
handler.addFilter(RedactingFilter())
logging.basicConfig(level=logging.INFO, handlers=[handler])

logging.info("connecting to postgresql://gis:[email protected]/gis")
# → connecting to postgresql://gis:***@db.internal/gis

And never let a credential into an exception message:

# leaks the password in every traceback
raise RuntimeError(f"cannot connect to {url}")

# safe
raise RuntimeError(f"cannot connect to {engine.url.render_as_string(hide_password=True)}")

Redact in your run records too — the lineage file is written to disk and often copied into tickets.

Handle secrets correctly in Docker

# ✗ wrong: baked into a layer, visible in `docker history` forever
ENV PGPASSWORD=Summer2026!

# ✗ wrong: the file stays in the layer even if a later RUN deletes it
COPY .env /app/.env

# ✓ right: mounted at build time only, never stored in a layer
RUN --mount=type=secret,id=pgpass \
    PGPASSWORD="$(cat /run/secrets/pgpass)" python3 scripts/fetch_reference_data.py

docker build --secret id=pgpass,src=./secrets/db_password.txt -t gis-pipeline .

docker run --rm \
  -e PGUSER -e PGDATABASE -e PGHOST \
  -e PGPASSWORD_FILE=/run/secrets/db_password \
  -v "$PWD/secrets/db_password.txt:/run/secrets/db_password:ro" \
  gis-pipeline

Note that -e PGUSER with no value passes the variable through from your shell without it appearing in the command line — where it would otherwise be visible in ps output and shell history.

Handle secrets correctly in CI

# .github/workflows/pipeline.yml
jobs:
  run:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Run pipeline
        env:
          PGUSER: ${{ secrets.PGUSER }}
          PGPASSWORD: ${{ secrets.PGPASSWORD }}
          PGDATABASE: gis
        run: python -m src.pipeline --config configs/daily.yml

GitHub masks registered secrets in logs automatically, but only the exact value — a base64-encoded or partially printed secret is not masked. Never echo one, and be careful with set -x in shell steps, which prints every expanded command.

Fail fast, with a message that helps

import sys

def check_secrets(required=("PGUSER", "PGPASSWORD", "PGDATABASE")) -> None:
    missing = [n for n in required
               if not os.environ.get(n) and not os.environ.get(f"{n}_FILE")]
    if missing:
        print(
            "missing credentials: " + ", ".join(missing) + "\n"
            "  local dev : copy .env.example to .env and fill it in\n"
            "  server    : set them in /srv/gis/job.env\n"
            "  CI        : add them as repository secrets",
            file=sys.stderr,
        )
        raise SystemExit(2)

check_secrets()

A job that refuses to start with a specific message beats one that fails forty minutes in with FATAL: password authentication failed.

Rotate, and make rotation cheap

import os
from datetime import datetime, timezone

def credential_age_days(env_var="PGPASSWORD_SET_AT") -> float | None:
    stamp = os.environ.get(env_var)
    if not stamp:
        return None
    set_at = datetime.fromisoformat(stamp)
    return (datetime.now(timezone.utc) - set_at).days

age = credential_age_days()
if age and age > 90:
    logging.warning("database credential is %d days old — rotate it", age)

Rotation is only realistic when nothing is hard-coded: one place to change, one restart, done. That is the practical argument for all of the above, quite apart from the security one.

Code examples

Example 1: a settings module worth copying

"""settings.py — one place that knows where configuration comes from."""
from dataclasses import dataclass
from pathlib import Path
import os

def _read(name: str, default: str | None = None, required: bool = False) -> str | None:
    file_var = os.environ.get(f"{name}_FILE")
    if file_var:
        return Path(file_var).read_text(encoding="utf-8").strip()
    value = os.environ.get(name, default)
    if required and not value:
        raise RuntimeError(f"missing required setting {name} (or {name}_FILE)")
    return value

@dataclass(frozen=True)
class Settings:
    pg_user: str
    pg_password: str
    pg_host: str
    pg_port: int
    pg_database: str
    data_root: Path
    s3_key: str | None
    s3_secret: str | None

    @classmethod
    def from_env(cls) -> "Settings":
        return cls(
            pg_user=_read("PGUSER", required=True),
            pg_password=_read("PGPASSWORD", required=True),
            pg_host=_read("PGHOST", "localhost"),
            pg_port=int(_read("PGPORT", "5432")),
            pg_database=_read("PGDATABASE", required=True),
            data_root=Path(_read("GIS_DATA_ROOT", "./data")).expanduser().resolve(),
            s3_key=_read("AWS_ACCESS_KEY_ID"),
            s3_secret=_read("AWS_SECRET_ACCESS_KEY"),
        )

    def __repr__(self) -> str:                     # safe to log
        return (f"Settings(pg_user={self.pg_user!r}, pg_host={self.pg_host!r}, "
                f"pg_database={self.pg_database!r}, data_root={self.data_root!r}, "
                f"pg_password=***, s3_secret={'set' if self.s3_secret else 'unset'})")

settings = Settings.from_env()
print(settings)          # no secret can escape through a print or a traceback

Overriding __repr__ is a small thing that prevents a large class of accident: dataclasses print all their fields by default, and that repr ends up in tracebacks and debug logs.

Example 2: connect to PostGIS without building a URL by hand

import geopandas as gpd
from sqlalchemy import create_engine
from sqlalchemy.engine import URL

def postgis_engine(settings):
    url = URL.create("postgresql+psycopg",
                     username=settings.pg_user, password=settings.pg_password,
                     host=settings.pg_host, port=settings.pg_port,
                     database=settings.pg_database)
    return create_engine(url, pool_pre_ping=True, connect_args={"connect_timeout": 10})

engine = postgis_engine(settings)
gdf = gpd.read_postgis("SELECT id, class, geom FROM parcels LIMIT 1000",
                       engine, geom_col="geom")
print(f"{len(gdf)} rows from {engine.url.render_as_string(hide_password=True)}")

For psql and GDAL command-line tools, a ~/.pgpass file with 0600 permissions keeps the password out of both the command line and the environment:

db.internal:5432:gis:gis_user:Summer2026!

Example 3: cloud storage credentials for GDAL

import os
import geopandas as gpd

# credentials come from the environment or the instance's role — never from code
for var in ("AWS_ACCESS_KEY_ID", "AWS_SECRET_ACCESS_KEY"):
    if not os.environ.get(var):
        print(f"{var} not set — relying on the instance role")

os.environ.setdefault("AWS_REGION", "eu-west-2")
os.environ.setdefault("GDAL_DISABLE_READDIR_ON_OPEN", "EMPTY_DIR")

gdf = gpd.read_file("/vsis3/my-bucket/parcels/parcels.gpkg")
print(len(gdf))

On a cloud VM, an attached instance role or workload identity removes the long-lived key entirely, which is strictly better than any way of storing one.

Example 4: scan the repository before you push

#!/usr/bin/env python3
"""scan_secrets.py — a crude pre-commit check for committed credentials."""
import re, subprocess, sys
from pathlib import Path

PATTERNS = {
    "postgres url": re.compile(r"postgres(?:ql)?://[^:\s]+:[^@\s]+@"),
    "aws key": re.compile(r"AKIA[0-9A-Z]{16}"),
    "private key": re.compile(r"-----BEGIN (?:RSA |EC |OPENSSH )?PRIVATE KEY-----"),
    "assigned secret": re.compile(r"(?i)(password|token|secret|api_key)\s*=\s*['\"][^'\"]{6,}"),
}

staged = subprocess.run(["git", "diff", "--cached", "--name-only"],
                        capture_output=True, text=True).stdout.split()
problems = []
for name in staged:
    path = Path(name)
    if not path.is_file() or path.suffix in {".png", ".svg", ".gpkg", ".tif"}:
        continue
    text = path.read_text(errors="ignore")
    for label, pattern in PATTERNS.items():
        for match in pattern.finditer(text):
            line = text[:match.start()].count("\n") + 1
            problems.append(f"{path}:{line}: possible {label}")

if problems:
    print("possible secrets in staged changes:", file=sys.stderr)
    for p in problems:
        print("  " + p, file=sys.stderr)
    sys.exit(1)

Wire it into .git/hooks/pre-commit, or use gitleaks or detect-secrets via pre-commit, which are far more thorough.

Explanation

A secret is not dangerous because it is written down; it is dangerous because of how many copies exist and how long they live. Hard-coding is the worst case on both counts: a value in the source tree is duplicated into every clone, every branch, every backup and every git object — and removing it later requires rewriting history, which never fully succeeds.

Checklist of secret-handling controls for an automated GIS job. — Seven controls, in the order they are usually skipped.

Environment variables solve the duplication problem, which is why they became the convention: the value lives in one place per environment, the code refers to it by name, and moving between laptop, server and CI needs no code change. They are not perfectly private — anything that can read /proc/<pid>/environ can read them, and they are inherited by every subprocess — but for most pipelines they are the right balance of safety and simplicity.

Secret files are a step tighter and are what container platforms actually provide. Docker mounts secrets under /run/secrets/, Kubernetes mounts them as files, and systemd has LoadCredential. Supporting the NAME_FILE convention alongside NAME costs six lines and makes the same code work in every one of those environments — which is why so many official images use it.

The leak path people underestimate is output. A connection string interpolated into a log line, an exception message, or a run record is a credential written to disk in a file nobody thinks of as sensitive, then copied into a ticket, an email and a chat thread. Building the URL only at the point of use, redacting in the logging pipeline, and overriding __repr__ on the settings object close that path at three levels — which is roughly the right number, because any one of them can be bypassed by a stray print.

Finally, all of this makes rotation possible, and rotation is the control that limits the damage when something does leak. If a password appears in exactly one place per environment, changing it is a two-minute job. If it is scattered through code, notebooks and images, rotation is a project — so it does not happen, and a leaked credential stays valid for years.

Edge cases or notes

Deleting a secret from a file does not remove it from git: It lives in history. Rotate the credential; do not rely on a follow-up commit.
docker history shows ENV values: Anything set with ENV in a Dockerfile is permanently visible in the image metadata.
Command lines are public: psql --password=... appears in ps output. Use ~/.pgpass or environment variables instead.
Notebook outputs persist: A printed token is saved in the .ipynb. Clear outputs before committing, or use nbstripout.
CI masking is literal: GitHub masks the exact secret value only. Encoded, split or partially printed secrets appear in plain text.
.env files get copied: Add .env to .dockerignore as well as .gitignore, or it lands in the build context.
Prefer roles to keys in the cloud: An instance role or workload identity issues short-lived credentials, which cannot be leaked in a durable form.

Internal links

FAQ

Where should credentials for a scheduled GIS job live?

In the environment of the job — an EnvironmentFile for systemd, a sourced env file for cron, repository secrets for CI — or in a file whose path is given by an environment variable. Never in the code or the config file that is committed.

Are environment variables secure enough?

For most pipelines, yes. They are readable by anything that can read the process's environment, so for stricter requirements use mounted secret files (NAME_FILE) or a secret manager that issues short-lived credentials.

How do I stop a password appearing in logs?

Build the connection URL only where it is used, log engine.url.render_as_string(hide_password=True), add a redacting logging.Filter, and override __repr__ on any settings object so it cannot print its own fields.

Is a `.env` file safe?

For local development, with .env in both .gitignore and .dockerignore from the start. Commit a .env.example with empty values so the required names are documented.

How do I pass secrets into a Docker build?

RUN --mount=type=secret,id=name with docker build --secret. Never ENV or COPY a secret — both persist in the image layers and show up in docker history.

What if a credential has already been committed?

Rotate it immediately. Removing the file in a later commit does not remove it from history, and any clone made before the removal still contains it.

Do I need a secret manager like Vault or AWS Secrets Manager?

Not for a single pipeline with a couple of credentials. They pay off when secrets are shared across teams and services, or when you need audited access and automatic rotation.

How to Handle Credentials and Secrets in an Automated GIS Job #

Problem statement #

Quick answer #

Where secrets can live #

Step-by-step solution #

Get secrets out of the code #

Load a .env for local development only #

Prefer secret files over environment variables where you can #

Keep secrets out of logs and tracebacks #

Handle secrets correctly in Docker #

Handle secrets correctly in CI #

Fail fast, with a message that helps #

Rotate, and make rotation cheap #

Code examples #

Example 1: a settings module worth copying #

Example 2: connect to PostGIS without building a URL by hand #

Example 3: cloud storage credentials for GDAL #

Example 4: scan the repository before you push #

Explanation #

Edge cases or notes #

Internal links #

FAQ #

Where should credentials for a scheduled GIS job live? #

Are environment variables secure enough? #

How do I stop a password appearing in logs? #

Is a .env file safe? #

How do I pass secrets into a Docker build? #

What if a credential has already been committed? #

Do I need a secret manager like Vault or AWS Secrets Manager? #