How to Handle Credentials and Secrets in an Automated GIS Job

Problem statement

The pipeline needs a PostGIS password, an S3 key pair and an API token. The quickest way to make it work is the one everybody regrets:

engine = create_engine("postgresql://gis_user:Summer2026!@db.internal:5432/gis")

Committed on Tuesday, pushed to a shared repository, copied into a Jupyter notebook, pasted into a Slack thread, then found by a scanner eighteen months later — long after that password was reused on three other systems. Even without a leak, hard-coded credentials mean the job cannot move between environments, and rotating the password means editing code.

The problems compound in an automated context:

• a credential in the source tree is in every clone and every git object, forever
• a scheduled job cannot answer an interactive password prompt
• a connection string leaks into logs, tracebacks, run records and error emails
• .env files get committed by accident, or copied into a Docker image layer
• a token pasted into a notebook ends up in the .ipynb output cells

Quick answer

Keep secrets out of code, out of logs, and out of images:

1. read credentials from environment variables, or from files whose path is in an environment variable
2. keep a .env for local development, and put it in .gitignore on day one
3. never log or print a connection string — build the URL only where it is used
4. in Docker and CI, use the platform's secret mechanism, not ENV or a COPY
5. fail with a clear message when a required secret is missing

import os
from sqlalchemy import create_engine
from sqlalchemy.engine import URL

def require_env(name: str) -> str:
    value = os.environ.get(name)
    if not value:
        raise RuntimeError(
            f"missing required environment variable {name}. "
            f"Set it in the environment, or in a .env file for local development."
        )
    return value

def make_engine():
    url = URL.create(
        "postgresql+psycopg",
        username=require_env("PGUSER"),
        password=require_env("PGPASSWORD"),      # never interpolated into a string
        host=os.environ.get("PGHOST", "localhost"),
        port=int(os.environ.get("PGPORT", 5432)),
        database=require_env("PGDATABASE"),
    )
    return create_engine(url)

engine = make_engine()
with engine.connect() as conn:
    conn.exec_driver_sql("SELECT 1")
print("connected to", engine.url.render_as_string(hide_password=True))
print("str(engine.url) is safe too:", str(engine.url))

connected to postgresql+psycopg://user:***@localhost:55433/gis
str(engine.url) is safe too: postgresql+psycopg://user:***@localhost:55433/gis

Run against a local PostGIS (set up in The data used on this page), the printed URL shows *** in place of the password — and so does plain str(engine.url) in SQLAlchemy 2, which is a useful safety net but not one to rely on in other libraries. URL.create handles escaping for you — a password containing @, / or # breaks a hand-built URL — and render_as_string(hide_password=True) is the only form that should ever reach a log.

Where secrets can live

Names in the repository, values elsewhere

Get secrets out of the code

The rule is simple: the repository contains the names of secrets, never their values.

# config.py — names and defaults only
REQUIRED_SECRETS = ("PGUSER", "PGPASSWORD", "PGDATABASE")
OPTIONAL_SETTINGS = {"PGHOST": "localhost", "PGPORT": "5432", "GIS_DATA_ROOT": "./data"}

# .env.example — committed, documents what is needed, contains nothing real
PGUSER=gis_user
PGPASSWORD=changeme
PGDATABASE=gis
PGHOST=db.internal
AWS_ACCESS_KEY_ID=
AWS_SECRET_ACCESS_KEY=

# .gitignore — add these before you create the real file
.env
.env.*
!.env.example
secrets/
*.pem
*.key

Load a .env for local development only

from pathlib import Path
import os

def load_dotenv(path: Path = Path(".env"), override: bool = False) -> int:
    """Minimal .env loader — real environment variables win by default."""
    if not path.exists():
        return 0
    loaded = 0
    for line in path.read_text(encoding="utf-8").splitlines():
        line = line.strip()
        if not line or line.startswith("#") or "=" not in line:
            continue
        key, value = line.split("=", 1)
        key, value = key.strip(), value.strip().strip('"').strip("'")
        if override or key not in os.environ:
            os.environ[key] = value
            loaded += 1
    return loaded


print("variables loaded from .env:", load_dotenv(), "| PGPASSWORD still from the real environment:",
      os.environ["PGPASSWORD"] == "guide")

variables loaded from .env: 0 | PGPASSWORD still from the real environment: True

Or use python-dotenv, which does the same with more edge cases handled:

from dotenv import load_dotenv
load_dotenv()                       # local only; in production the env is already set

Real environment variables must take precedence over the file, so a server's configuration is never overridden by a stray .env that got copied along with the code. The .env in this example holds a different password from the environment, and none of its values replaced one that was already set.

Prefer secret files over environment variables where you can

Environment variables are visible in /proc/<pid>/environ, in docker inspect, and in any subprocess. A file with 0600 permissions is tighter, and it is the mechanism Docker and Kubernetes secrets use.

import os
from pathlib import Path

def read_secret(name: str, default=None) -> str:
    """Read NAME_FILE if set (Docker/K8s style), else NAME, else default."""
    file_var = os.environ.get(f"{name}_FILE")
    if file_var:
        path = Path(file_var)
        if not path.exists():
            raise RuntimeError(f"{name}_FILE points at {path}, which does not exist")
        return path.read_text(encoding="utf-8").strip()
    value = os.environ.get(name, default)
    if value is None:
        raise RuntimeError(f"missing secret: set {name} or {name}_FILE")
    return value

os.environ["PGPASSWORD_FILE"] = "secrets/db_password.txt"            # written by the setup, mode 0600
password = read_secret("PGPASSWORD")
print("read from file:", password == Path("secrets/db_password.txt").read_text().strip(),
      "| file mode:", oct(Path("secrets/db_password.txt").stat().st_mode & 0o777))
del os.environ["PGPASSWORD_FILE"]

read from file: True | file mode: 0o600

This one function supports plain env vars, Docker secrets (/run/secrets/...), Kubernetes mounted secrets and systemd credentials without any change to the call site.

Keep secrets out of logs and tracebacks

This is where most real leaks happen — not from the repository, but from an error email.

import logging, re

SECRET_PATTERN = re.compile(
    r"(password|passwd|pwd|token|secret|api[_-]?key|authorization)"
    r"(\s*[=:]\s*|%3D)([^\s&'\"]+)", re.IGNORECASE)

class RedactingFilter(logging.Filter):
    def filter(self, record: logging.LogRecord) -> bool:
        if isinstance(record.msg, str):
            record.msg = SECRET_PATTERN.sub(r"\1=***", record.msg)
            record.msg = re.sub(r"(://[^:/@\s]+):([^@\s]+)@", r"\1:***@", record.msg)
        return True

handler = logging.StreamHandler()
handler.addFilter(RedactingFilter())
logging.basicConfig(level=logging.INFO, handlers=[handler])

logging.info("connecting to postgresql://gis:hunter2@db.internal/gis")
# → connecting to postgresql://gis:***@db.internal/gis

Test it — including the way most logging calls are actually written, with the URL passed as an argument:

import io

captured = io.StringIO()
test_handler = logging.StreamHandler(captured)
test_handler.addFilter(RedactingFilter())
test_log = logging.getLogger("redaction-test")
test_log.addHandler(test_handler)
test_log.propagate = False

import contextlib

url = "postgresql://gis:hunter2@db.internal/gis"
errors = io.StringIO()
with contextlib.redirect_stderr(errors):                       # where logging reports its own failures
    test_log.warning("connecting to " + url)                   # the secret is inside msg
    test_log.warning("connecting to %s", url)                  # the secret is inside args
    test_log.warning("retrying with password=%s", "hunter2")
print(captured.getvalue())
print("stderr:", errors.getvalue().strip().splitlines()[0], "...", errors.getvalue().strip().splitlines()[-1])

connecting to postgresql://gis:***@db.internal/gis
connecting to postgresql://gis:hunter2@db.internal/gis

stderr: --- Logging error --- ... Arguments: ('hunter2',)

The filter only rewrites record.msg, the format string, and that goes wrong in two ways. With %s arguments — the style the logging documentation recommends — the secret is in record.args and the second line passes it straight through. And the third line is worse: the filter turned password=%s into password=***, leaving an argument with no placeholder, so formatting failed, the message never reached the log — and logging's own error report on stderr printed the arguments, password included. Redact the fully formatted message instead:

class RedactingFilter(logging.Filter):
    def filter(self, record: logging.LogRecord) -> bool:
        message = record.getMessage()                               # msg % args, formatted
        message = SECRET_PATTERN.sub(r"\1=***", message)
        record.msg = re.sub(r"(://[^:/@\s]+):([^@\s]+)@", r"\1:***@", message)
        record.args = None
        return True


captured.truncate(0); captured.seek(0)
test_handler.filters.clear()
test_handler.addFilter(RedactingFilter())
test_log.warning("connecting to %s", url)
test_log.warning("retrying with password=%s", "hunter2")
print(captured.getvalue())

connecting to postgresql://gis:***@db.internal/gis
retrying with password=***

And never let a credential into an exception message:

# leaks the password in every traceback
raise RuntimeError(f"cannot connect to {url}")

# safe
raise RuntimeError(f"cannot connect to {engine.url.render_as_string(hide_password=True)}")

Redact in your run records too — the lineage file is written to disk and often copied into tickets.

Handle secrets correctly in Docker

# ✗ wrong: baked into a layer, visible in `docker history` forever
ENV PGPASSWORD=Summer2026!

# ✗ wrong: the file stays in the layer even if a later RUN deletes it
COPY .env /app/.env

# ✓ right: mounted at build time only, never stored in a layer
RUN --mount=type=secret,id=pgpass \
    PGPASSWORD="$(cat /run/secrets/pgpass)" python3 scripts/fetch_reference_data.py

docker build --secret id=pgpass,src=./secrets/db_password.txt -t gis-pipeline .

docker run --rm \
  -e PGUSER -e PGDATABASE -e PGHOST \
  -e PGPASSWORD_FILE=/run/secrets/db_password \
  -v "$PWD/secrets/db_password.txt:/run/secrets/db_password:ro" \
  gis-pipeline

Note that -e PGUSER with no value passes the variable through from your shell without it appearing in the command line — where it would otherwise be visible in ps output and shell history.

The three Dockerfile lines above, built for real from alpine and inspected:

import shutil
import tarfile

Path("dk").mkdir(exist_ok=True)
Path("dk/.env").write_text("PGPASSWORD=Summer2026!\n")
Path("dk/pgpass.txt").write_text("Summer2026!")
dockerfiles = {
    "env": "FROM alpine:latest\nENV PGPASSWORD=Summer2026!\n",
    "copy": "FROM alpine:latest\nCOPY .env /app/.env\nRUN rm /app/.env\n",
    "mount": "FROM alpine:latest\nRUN --mount=type=secret,id=pgpass wc -c < /run/secrets/pgpass > /secret_length\n",
}
for name, text in dockerfiles.items():
    Path(f"dk/Dockerfile.{name}").write_text(text)
    subprocess.run(["docker", "build", "-q", "-f", f"dk/Dockerfile.{name}", "--secret", "id=pgpass,src=dk/pgpass.txt",
                    "-t", f"sw-secrets-{name}", "dk"], check=True, capture_output=True)

history = subprocess.run(["docker", "history", "--no-trunc", "--format", "{{.CreatedBy}}", "sw-secrets-env"],
                         capture_output=True, text=True).stdout
print("ENV   in docker history:", "Summer2026!" in history)

subprocess.run(["docker", "save", "sw-secrets-copy", "-o", "dk/copy.tar"], check=True)
with tarfile.open("dk/copy.tar") as image:
    layers = [m for m in image.getmembers() if m.isfile() and m.size > 0]
    found = [m.name for m in layers if tarfile.is_tarfile(image.extractfile(m))
             and "app/.env" in tarfile.open(fileobj=image.extractfile(m)).getnames()]
print("COPY  then RUN rm — .env still in a layer:", bool(found))

history = subprocess.run(["docker", "history", "--no-trunc", "--format", "{{.CreatedBy}}", "sw-secrets-mount"],
                         capture_output=True, text=True).stdout
inside = subprocess.run(["docker", "run", "--rm", "sw-secrets-mount", "sh", "-c", "cat /secret_length; ls /run/secrets"],
                        capture_output=True, text=True)
print("mount in docker history:", "Summer2026!" in history, "| secret was", inside.stdout.split()[0],
      "bytes at build time | /run/secrets in the image:", "No such file" not in inside.stderr)
subprocess.run(["docker", "rmi", "-f", "sw-secrets-env", "sw-secrets-copy", "sw-secrets-mount"], capture_output=True)

ENV   in docker history: True
COPY  then RUN rm — .env still in a layer: True
mount in docker history: False | secret was 11 bytes at build time | /run/secrets in the image: False

The ENV value is in the image's history for anyone who can pull it; the deleted .env is still inside the layer that COPY created, recoverable with docker save; the secret mount was readable during the build — 11 bytes — and left no trace in the history or the image.

Handle secrets correctly in CI

# .github/workflows/pipeline.yml
jobs:
  run:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Run pipeline
        env:
          PGUSER: ${{ secrets.PGUSER }}
          PGPASSWORD: ${{ secrets.PGPASSWORD }}
          PGDATABASE: gis
        run: python -m src.pipeline --config configs/daily.yml

GitHub masks registered secrets in logs automatically, but only the exact value — a base64-encoded or partially printed secret is not masked. Never echo one, and be careful with set -x in shell steps, which prints every expanded command.

Fail fast, with a message that helps

import sys

def check_secrets(required=("PGUSER", "PGPASSWORD", "PGDATABASE")) -> None:
    missing = [n for n in required
               if not os.environ.get(n) and not os.environ.get(f"{n}_FILE")]
    if missing:
        print(
            "missing credentials: " + ", ".join(missing) + "\n"
            "  local dev : copy .env.example to .env and fill it in\n"
            "  server    : set them in /srv/gis/job.env\n"
            "  CI        : add them as repository secrets",
            file=sys.stderr,
        )
        raise SystemExit(2)

try:
    check_secrets(required=("PGUSER", "PGPASSWORD", "PGDATABASE", "MAPBOX_TOKEN"))
except SystemExit as stop:
    print("exit code", stop.code)

exit code 2

A job that refuses to start with a specific message beats one that fails forty minutes in with FATAL: password authentication failed.

Rotate, and make rotation cheap

import os
from datetime import datetime, timezone

def credential_age_days(env_var="PGPASSWORD_SET_AT", now=None) -> float | None:
    stamp = os.environ.get(env_var)
    if not stamp:
        return None
    set_at = datetime.fromisoformat(stamp)
    if set_at.tzinfo is None:                     # "2026-06-01" has no zone: treat it as UTC
        set_at = set_at.replace(tzinfo=timezone.utc)
    return ((now or datetime.now(timezone.utc)) - set_at).days


checked = datetime(2026, 10, 1, 6, 0, tzinfo=timezone.utc)   # a fixed "now", so the output reproduces
for stamp in ["2026-06-01", "2026-06-01T09:00:00+01:00"]:
    os.environ["PGPASSWORD_SET_AT"] = stamp
    age = credential_age_days(now=checked)
    print(f"set {stamp}: {age} days old{' — rotate it' if age > 90 else ''}")

set 2026-06-01: 122 days old — rotate it
set 2026-06-01T09:00:00+01:00: 121 days old — rotate it

The tzinfo line matters: someone will write the date as 2026-06-01, fromisoformat returns a naive datetime for it, and subtracting a naive datetime from an aware one raises TypeError — so without it, the rotation check crashes on the most natural way to fill it in.

Rotation is only realistic when nothing is hard-coded: one place to change, one restart, done. That is the practical argument for all of the above, quite apart from the security one.

Credentials for a throwaway PostGIS

The data used on this page

The credentials on this page belong to a throwaway PostGIS started by the setup in Docker, on port 55433, user user, password guide — the only real secret here, and one that protects nothing. The setup also loads the Land Registry's INSPIRE parcels for Rutland into it as a table parcels, writes a .env holding a different password (to show which one wins), and a secret file with mode 0600.

import io
import logging
import os
import re
import subprocess
import sys
import time
import zipfile
from datetime import datetime, timezone
from pathlib import Path

import geopandas as gpd
import pyogrio
import requests
from sqlalchemy import create_engine

os.environ.update(PGUSER="user", PGPASSWORD="guide", PGDATABASE="gis", PGHOST="localhost", PGPORT="55433")
if os.system("docker inspect guide-postgis >/dev/null 2>&1") != 0:
    os.system("docker run -d --name guide-postgis -e POSTGRES_USER=user -e POSTGRES_PASSWORD=guide "
              "-e POSTGRES_DB=gis -p 127.0.0.1:55433:5432 postgis/postgis:16-3.4 >/dev/null")
    time.sleep(10)
pyogrio.set_gdal_config_options({"GML_DOWNLOAD_SCHEMA": "NO"})
if not Path("parcels.gpkg").exists():
    reply = requests.Session().get("https://use-land-property-data.service.gov.uk/datasets/inspire/download/"
                                   "Rutland.zip", timeout=600)
    reply.raise_for_status()
    zipfile.ZipFile(io.BytesIO(reply.content)).extract("Land_Registry_Cadastral_Parcels.gml")
    gpd.read_file("Land_Registry_Cadastral_Parcels.gml").to_file("parcels.gpkg")
gpd.read_file("parcels.gpkg").rename_geometry("geom").to_postgis(
    "parcels", create_engine("postgresql+psycopg://user@localhost:55433/gis"), if_exists="replace", index=False)

Path(".env").write_text("PGUSER=gis_user\nPGPASSWORD=from-the-dotenv-file\nPGDATABASE=gis\n")
Path("secrets").mkdir(exist_ok=True)
Path("secrets/db_password.txt").write_text("guide\n")
os.chmod("secrets/db_password.txt", 0o600)
print("PostGIS on 55433; parcels table loaded")

PostGIS on 55433; parcels table loaded

Example 1: a settings module worth copying

"""settings.py — one place that knows where configuration comes from."""
from dataclasses import dataclass
from pathlib import Path
import os

def _read(name: str, default: str | None = None, required: bool = False) -> str | None:
    file_var = os.environ.get(f"{name}_FILE")
    if file_var:
        return Path(file_var).read_text(encoding="utf-8").strip()
    value = os.environ.get(name, default)
    if required and not value:
        raise RuntimeError(f"missing required setting {name} (or {name}_FILE)")
    return value

@dataclass(frozen=True)
class Settings:
    pg_user: str
    pg_password: str
    pg_host: str
    pg_port: int
    pg_database: str
    data_root: Path
    s3_key: str | None
    s3_secret: str | None

    @classmethod
    def from_env(cls) -> "Settings":
        return cls(
            pg_user=_read("PGUSER", required=True),
            pg_password=_read("PGPASSWORD", required=True),
            pg_host=_read("PGHOST", "localhost"),
            pg_port=int(_read("PGPORT", "5432")),
            pg_database=_read("PGDATABASE", required=True),
            data_root=Path(_read("GIS_DATA_ROOT", "./data")).expanduser().resolve(),
            s3_key=_read("AWS_ACCESS_KEY_ID"),
            s3_secret=_read("AWS_SECRET_ACCESS_KEY"),
        )

    def __repr__(self) -> str:                     # safe to log
        return (f"Settings(pg_user={self.pg_user!r}, pg_host={self.pg_host!r}, "
                f"pg_database={self.pg_database!r}, data_root={self.data_root!r}, "
                f"pg_password=***, s3_secret={'set' if self.s3_secret else 'unset'})")

os.environ["GIS_DATA_ROOT"] = "/srv/gis/data"
settings = Settings.from_env()
print(settings)          # no secret can escape through a print or a traceback

Settings(pg_user='user', pg_host='localhost', pg_database='gis', data_root=PosixPath('/srv/gis/data'), pg_password=***, s3_secret=unset)

Overriding __repr__ is a small thing that prevents a large class of accident: dataclasses print all their fields by default, and that repr ends up in tracebacks and debug logs.

Example 2: connect to PostGIS without building a URL by hand

import geopandas as gpd
from sqlalchemy import create_engine
from sqlalchemy.engine import URL

def postgis_engine(settings):
    url = URL.create("postgresql+psycopg",
                     username=settings.pg_user, password=settings.pg_password,
                     host=settings.pg_host, port=settings.pg_port,
                     database=settings.pg_database)
    return create_engine(url, pool_pre_ping=True, connect_args={"connect_timeout": 10})

engine = postgis_engine(settings)
gdf = gpd.read_postgis('SELECT "INSPIREID" AS id, geom FROM parcels LIMIT 1000',
                       engine, geom_col="geom")
print(f"{len(gdf)} rows from {engine.url.render_as_string(hide_password=True)}")

1000 rows from postgresql+psycopg://user:***@localhost:55433/gis

For psql and GDAL command-line tools, a ~/.pgpass file with 0600 permissions keeps the password out of both the command line and the environment:

db.internal:5432:gis:gis_user:Summer2026!

Example 3: cloud storage credentials for GDAL

GDAL reads S3 credentials from the same environment variables as the AWS tools, or from an instance role, so nothing about them belongs in code. The Copernicus elevation tiles sit in a public bucket, which shows both sides — a request with no credentials at all, and one that says explicitly that none are needed:

import rasterio

TILE = ("/vsis3/copernicus-dem-30m/Copernicus_DSM_COG_10_N52_00_W001_00_DEM/"
        "Copernicus_DSM_COG_10_N52_00_W001_00_DEM.tif")
for var in ("AWS_ACCESS_KEY_ID", "AWS_SECRET_ACCESS_KEY", "AWS_PROFILE"):
    if not os.environ.get(var):
        print(f"{var} not set")

for options in ({}, {"AWS_NO_SIGN_REQUEST": "YES"}):
    try:
        with rasterio.Env(GDAL_DISABLE_READDIR_ON_OPEN="EMPTY_DIR", AWS_REGION="eu-central-1", **options):
            with rasterio.open(TILE) as src:
                print(options or "no options", "-> opened", src.width, "x", src.height)
    except rasterio.errors.RasterioIOError as err:
        print(options or "no options", "->", str(err)[:110])

AWS_ACCESS_KEY_ID not set
AWS_SECRET_ACCESS_KEY not set
AWS_PROFILE not set
no options -> InvalidCredentials: No valid AWS credentials found. For authenticated requests, you need to set AWS_SECRET_ACC
{'AWS_NO_SIGN_REQUEST': 'YES'} -> opened 2400 x 3600

On a cloud VM, an attached instance role or workload identity removes the long-lived key entirely, which is strictly better than any way of storing one.

Example 4: scan the repository before you push

%%writefile scan_secrets.py
#!/usr/bin/env python3
"""scan_secrets.py — a crude pre-commit check for committed credentials."""
import re, subprocess, sys
from pathlib import Path

PATTERNS = {
    "postgres url": re.compile(r"postgres(?:ql)?://[^:\s]+:[^@\s]+@"),
    "aws key": re.compile(r"AKIA[0-9A-Z]{16}"),
    "private key": re.compile(r"-----BEGIN (?:RSA |EC |OPENSSH )?PRIVATE KEY-----"),
    "assigned secret": re.compile(r"(?i)(password|token|secret|api_key)\s*=\s*['\"][^'\"]{6,}"),
}

staged = subprocess.run(["git", "diff", "--cached", "--name-only"],
                        capture_output=True, text=True).stdout.split()
problems = []
for name in staged:
    path = Path(name)
    if not path.is_file() or path.suffix in {".png", ".svg", ".gpkg", ".tif"}:
        continue
    text = path.read_text(errors="ignore")
    for label, pattern in PATTERNS.items():
        for match in pattern.finditer(text):
            line = text[:match.start()].count("\n") + 1
            problems.append(f"{path}:{line}: possible {label}")

if problems:
    print("possible secrets in staged changes:", file=sys.stderr)
    for p in problems:
        print("  " + p, file=sys.stderr)
    sys.exit(1)

Tried in a scratch repository, with AWS's published example key (AKIAIOSFODNN7EXAMPLE, which is not a real key) and a connection string staged in a config file:

Path("scratch_repo").mkdir(exist_ok=True)
Path("scratch_repo/settings.py").write_text('DB = "postgresql://gis:hunter2@db.internal/gis"\n'
                                            'AWS_KEY = "AKIAIOSFODNN7EXAMPLE"\npassword = "hunter2-longer"\n')
Path("scratch_repo/scan_secrets.py").write_text(Path("scan_secrets.py").read_text())
for cmd in (["git", "init", "-q"], ["git", "add", "settings.py"]):
    subprocess.run(cmd, cwd="scratch_repo", check=True)
scan = subprocess.run([sys.executable, "scan_secrets.py"], cwd="scratch_repo", capture_output=True, text=True)
print("exit", scan.returncode)
print(scan.stderr)

exit 1
possible secrets in staged changes:
  settings.py:1: possible postgres url
  settings.py:2: possible aws key
  settings.py:3: possible assigned secret

Wire it into .git/hooks/pre-commit, or use gitleaks or detect-secrets via pre-commit, which are far more thorough.

Copies and lifetimes are the danger

A secret is not dangerous because it is written down; it is dangerous because of how many copies exist and how long they live. Hard-coding is the worst case on both counts: a value in the source tree is duplicated into every clone, every branch, every backup and every git object — and removing it later requires rewriting history, which never fully succeeds.

Environment variables solve the duplication problem, which is why they became the convention: the value lives in one place per environment, the code refers to it by name, and moving between laptop, server and CI needs no code change. They are not perfectly private — anything that can read /proc/<pid>/environ can read them, and they are inherited by every subprocess — but for most pipelines they are the right balance of safety and simplicity.

Secret files are a step tighter and are what container platforms actually provide. Docker mounts secrets under /run/secrets/, Kubernetes mounts them as files, and systemd has LoadCredential. Supporting the NAME_FILE convention alongside NAME costs six lines and makes the same code work in every one of those environments — which is why so many official images use it.

The leak path people underestimate is output. A connection string interpolated into a log line, an exception message, or a run record is a credential written to disk in a file nobody thinks of as sensitive, then copied into a ticket, an email and a chat thread. Building the URL only at the point of use, redacting in the logging pipeline, and overriding __repr__ on the settings object close that path at three levels — which is roughly the right number, because any one of them can be bypassed by a stray print.

Finally, all of this makes rotation possible, and rotation is the control that limits the damage when something does leak. If a password appears in exactly one place per environment, changing it is a two-minute job. If it is scattered through code, notebooks and images, rotation is a project — so it does not happen, and a leaked credential stays valid for years.

Secrets that git remembers

• Deleting a secret from a file does not remove it from git: It lives in history. Rotate the credential; do not rely on a follow-up commit.
• docker history shows ENV values: Anything set with ENV in a Dockerfile is permanently visible in the image metadata.
• Command lines are public: psql --password=... appears in ps output. Use ~/.pgpass or environment variables instead.
• Notebook outputs persist: A printed token is saved in the .ipynb. Clear outputs before committing, or use nbstripout.
• CI masking is literal: GitHub masks the exact secret value only. Encoded, split or partially printed secrets appear in plain text.
• .env files get copied: Add .env to .dockerignore as well as .gitignore, or it lands in the build context.
• Prefer roles to keys in the cloud: An instance role or workload identity issues short-lived credentials, which cannot be leaked in a durable form.

Internal links

• How to Connect GeoPandas to PostGIS
• How to Containerise a Python GIS Pipeline with Docker
• How to Run a Python GIS Pipeline in CI with GitHub Actions
• How to Drive a GIS Pipeline from a YAML Config File in Python
• Python GIS Script Works Manually but Not from Cron: How to Fix It
• How to Record Run Metadata and Data Lineage in a GIS Pipeline

FAQ

Where should credentials for a scheduled GIS job live?

In the environment of the job — an EnvironmentFile for systemd, a sourced env file for cron, repository secrets for CI — or in a file whose path is given by an environment variable. Never in the code or the config file that is committed.

Are environment variables secure enough?

For most pipelines, yes. They are readable by anything that can read the process's environment, so for stricter requirements use mounted secret files (NAME_FILE) or a secret manager that issues short-lived credentials.

How do I stop a password appearing in logs?

Build the connection URL only where it is used, log engine.url.render_as_string(hide_password=True), add a redacting logging.Filter, and override __repr__ on any settings object so it cannot print its own fields.

Is a .env file safe?

For local development, with .env in both .gitignore and .dockerignore from the start. Commit a .env.example with empty values so the required names are documented.

How do I pass secrets into a Docker build?

RUN --mount=type=secret,id=name with docker build --secret. Never ENV or COPY a secret — both persist in the image layers and show up in docker history.

What if a credential has already been committed?

Rotate it immediately. Removing the file in a later commit does not remove it from history, and any clone made before the removal still contains it.

Do I need a secret manager like Vault or AWS Secrets Manager?

Not for a single pipeline with a couple of credentials. They pay off when secrets are shared across teams and services, or when you need audited access and automatic rotation.