How to Set Up Jupyter for Python GIS Work

On this page

Problem statement

Jupyter is where most spatial analysis actually happens, and the default configuration fights you at every step.

A map renders at 640 Γ— 480 pixels β€” matplotlib's default 6.4 Γ— 4.8 inches at 100 dpi β€” too small to see a city's worth of parcels:

gdf.plot()

Then the kernel dies with no message, because a cell loaded a file bigger than the memory left and nothing warned you. Then a notebook that worked yesterday fails today, because someone pip installed into the wrong environment. Then a colleague opens your notebook and sees no output, because the interactive map needed an extension they do not have. Then a code review is unreadable, because the diff is thousands of lines of base64 PNG.

None of this is Jupyter being bad. It is a set of defaults chosen for teaching Python, applied to work involving large binary data and image output.

Quick answer

pixi init gis-notebooks && cd gis-notebooks
pixi add python=3.12 geopandas rasterio matplotlib jupyterlab ipykernel \
          folium contextily mapclassify pyogrio nbstripout
pixi run jupyter lab
# the first cell of every spatial notebook
import warnings
import matplotlib.pyplot as plt
import geopandas as gpd
import pandas as pd

%matplotlib inline
%config InlineBackend.figure_format = "retina"

plt.rcParams.update({"figure.figsize": (10, 10), "figure.dpi": 96,
                     "savefig.dpi": 200, "savefig.bbox": "tight",
                     "axes.grid": False})
pd.set_option("display.max_columns", 50, "display.width", 160)
gpd.options.display_precision = 4
The settings a spatial notebook environment needs. six settings, six daily annoyances removedA kernel registered per projectsys.executablefigsize and retina for readable mapsrcParamsHeader check before readingread_infonbstripout so diffs are reviewablegit filterPROJ_NETWORK and GDAL_CACHEMAXset before importLogic in .py files, narrative in the notebookautoreload 2The defaults were chosen for teaching Python, not for spatial data spatialworkflow.io
Six settings. Each removes a specific daily annoyance.
Problem Fix
tiny maps plt.rcParams["figure.figsize"]
blurry maps on a high-DPI screen %config InlineBackend.figure_format = "retina"
kernel dies silently check the file size before reading
wrong environment register a named kernel per project
unreadable diffs nbstripout
lost outputs for colleagues export to HTML, or use jupyter-scatter/static images

What you get β€” from 1. One environment per project, registered as a kernel

~/WebstormProjects/spatialworkflow_11ty/.cache/gisenv/bin/python
1.1.4 ~/WebstormProjects/spatialworkflow_11ty/.cache/gisenv/lib/python3.14/site-packages/geopandas/__init__.py

A kernel per project, readable maps, reviewable notebooks

1. One environment per project, registered as a kernel

The most common Jupyter problem is not Jupyter's fault: pip install geopandas in a terminal installs into whatever environment is active there, which is often not the one the kernel is running.

pixi add ipykernel
pixi run python -m ipykernel install --user \
    --name gis-project --display-name "Python (gis-project)"
# verify from inside the notebook
import sys
from pathlib import Path

import geopandas as gpd

home = str(Path.home())
print(sys.executable.replace(home, "~"))
print(gpd.__version__, gpd.__file__.replace(home, "~"))
~/WebstormProjects/spatialworkflow_11ty/.cache/gisenv/bin/python
1.1.4 ~/WebstormProjects/spatialworkflow_11ty/.cache/gisenv/lib/python3.14/site-packages/geopandas/__init__.py

If sys.executable is not inside your project, the kernel is wrong. Change it with Kernel β†’ Change Kernel rather than installing packages until it works.

Install from inside the notebook when you must, using the kernel's own interpreter:

import sys
!{sys.executable} -m pip install -q contextily      # -q: silent unless something changes

!pip install alone uses whatever pip is first on PATH, which is the root cause of most "I installed it and it still says ModuleNotFoundError".

2. Make maps readable

import matplotlib.pyplot as plt

%matplotlib inline
%config InlineBackend.figure_format = "retina"      # 2x resolution on HiDPI screens

plt.rcParams.update({
    "figure.figsize": (10, 10),      # the default 6.4 x 4.8 is too small for maps
    "figure.dpi": 96,                # screen
    "savefig.dpi": 200,              # export
    "savefig.bbox": "tight",
    "savefig.facecolor": "white",    # not transparent β€” dark viewers render it black
    "axes.grid": False,
    "axes.spines.top": False,
    "axes.spines.right": False,
    "font.size": 11,
})

figure_format = "retina" doubles the raster resolution of inline plots, which makes a real difference on any modern display. "svg" is sharper still and becomes slow beyond a few thousand features, since every polygon becomes a path element.

%config InlineBackend.figure_format = "svg"     # crisp, but only for small layers

For a quick look at a large layer, plot a sample rather than the whole thing:

gdf = gpd.read_file("parcels.gpkg")
ax = gdf.sample(min(5000, len(gdf)), random_state=0).plot(figsize=(10, 10))
ax.set_title(f"{len(gdf):,} features (5,000 sampled)")

3. Check before you read

Inspecting a file and estimating memory before loading it. a dead kernel is the OOM killer, and Jupyter cannot say so1read_info β€” header onlymilliseconds2Allow a margin over the file sizethen measure it3Compare against free memorypsutil4Read rows=, bbox= or columns=not the whole fileThe Leeds parcels: 190 MB on disk, about 214 MB once loaded β€” measure yours spatialworkflow.io
A header read costs milliseconds and prevents the kernel dying on a file bigger than the memory left.

A dead kernel with no message is almost always the OOM killer. Jupyter cannot report it, because the process was terminated without warning.

import pyogrio
from pathlib import Path

def peek(path, layer=None):
    """Header-only inspection, plus a memory estimate."""
    path = Path(path)
    info = pyogrio.read_info(path, layer=layer) if layer else pyogrio.read_info(path)
    size_mb = (path.stat().st_size / 1e6 if path.is_file()
               else sum(f.stat().st_size for f in path.rglob("*")) / 1e6)
    estimate = size_mb * 2.5          # a deliberately pessimistic allowance for Shapely objects
    print(f"{path.name}")
    print(f"  {info['features']:,} features, {len(info['fields'])} fields")
    print(f"  {info['geometry_type']}, {info['crs']}")
    print(f"  {size_mb:,.0f} MB on disk β†’ roughly {estimate:,.0f} MB in memory")
    try:
        import psutil
        available = psutil.virtual_memory().available / 1e6
        print(f"  {available:,.0f} MB available"
              + ("  ⚠ read a subset instead" if estimate > available * 0.6 else "  βœ“"))
    except ImportError:
        pass
    return info

peek("parcels.gpkg")
parcels.gpkg
  293,729 features, 6 fields
  Polygon, EPSG:27700
  190 MB on disk β†’ roughly 476 MB in memory
  29,124 MB available  βœ“

The Γ—2.5 is a deliberately pessimistic allowance, not a measurement. Loading all 293,729 parcels, with their id column, in Example 2 added about 214 MB to the process β€” barely more than the 190 MB file. The ratio depends on the data and the format, so measure it once for the files you work with and use your own number.

Then read what you actually need:

sample = gpd.read_file("parcels.gpkg", rows=5_000)                    # a look
area = gpd.read_file("parcels.gpkg", bbox=(428_000, 432_000, 432_000, 435_000))   # central Leeds
slim = gpd.read_file("parcels.gpkg", columns=["INSPIREID"])
print(f"{len(sample):,} sampled, {len(area):,} in the box, {len(slim):,} with one column")
5,000 sampled, 10,588 in the box, 293,729 with one column

Watch memory as you go. The classic tool is memory_profiler's %memit magic, but the package is unmaintained and imports distutils, which Python 3.12 removed β€” on this page's Python 3.14, %load_ext memory_profiler fails with ModuleNotFoundError: No module named 'distutils'. The process's resident memory, from psutil, gives the same increment with nothing to break:

import os

import psutil


def rss_mib():
    return psutil.Process(os.getpid()).memory_info().rss / 2**20


before = rss_mib()
first_100k = gpd.read_file("parcels.gpkg", rows=100_000)
print(f"reading 100,000 parcels: +{rss_mib() - before:,.0f} MiB (now {rss_mib():,.0f} MiB)")
reading 100,000 parcels: +48 MiB (now 974 MiB)

4. Keep notebooks reviewable

A committed notebook contains its outputs, which for spatial work means megabytes of base64 PNG in the diff.

pixi add nbstripout
pixi run nbstripout --install          # installs a git filter for this repo
git diff notebooks/analysis.ipynb      # now shows code changes only

nbstripout strips outputs on commit and leaves your working copy untouched, so you keep the rendered maps locally and the repository stays reviewable.

When the output is the deliverable, export rather than committing it:

jupyter nbconvert --to html --no-input notebooks/analysis.ipynb
jupyter nbconvert --execute --to html notebooks/analysis.ipynb    # fresh run

--no-input produces a report with figures and prose and no code, which is usually what a non-technical reader wants.

5. Interactive maps, and their cost

wards = gpd.read_file("wards.gpkg")
m = wards.explore(column="population", scheme="quantiles", k=5, cmap="YlOrRd",
                  tiles="CartoDB positron", tooltip=["WD24NM", "population"])
print(f"{len(m.get_root().render()) / 1e6:.2f} MB of HTML for {len(wards)} wards")
2.15 MB of HTML for 33 wards

GeoDataFrame.explore() builds a Folium map, which embeds the whole layer as GeoJSON in the notebook. That is excellent for a few thousand features and fatal beyond a few tens of thousands:

def safe_explore(gdf, max_features=5_000, max_mb=5, **kwargs):
    if len(gdf) > max_features:                       # decide on the count first: serialising is slow
        print(f"{len(gdf):,} features is too many for an interactive map β€” sampling {max_features:,}")
        gdf = gdf.sample(max_features, random_state=0)
    payload_mb = len(gdf.to_json(default=str)) / 1e6  # default=str: to_json cannot serialise Timestamps
    if payload_mb > max_mb:
        print(f"⚠ {payload_mb:.1f} MB of GeoJSON β€” simplify the geometry before sharing this")
    return gdf.explore(**kwargs)

parcel_map = safe_explore(gdf.assign(year=gdf["VALIDFROM"].dt.year), column="year")
293,729 features is too many for an interactive map β€” sampling 5,000

The size limits and the reasons behind them are in my Folium map is blank.

Note that an explore() map lives in the notebook's output. Strip outputs with nbstripout and it disappears for anyone who opens the file β€” which is correct behaviour, and worth knowing before a colleague reports a blank notebook.

How big an explore() map gets: HTML size and build time for Leeds wards and samples of Leeds parcels. what explore() puts in the notebooklayerfeaturesHTMLbuildLeeds wards, fullresolution332.1 MB2.0 sLeeds parcels, sample5,0006.9 MB0.7 sLeeds parcels, sample50,00068 MB7.2 sLeeds parcels, all293,729260 MB of GeoJSONnot attemptedvertices, not features, set the size β€” 33 detailed wards outweigh a thousand simple parcelsAll of it is embedded in the notebook file, and in every copy of it spatialworkflow.io
Measured on this page’s layers. Simplify or sample before you explore.

6. Set the environment variables that matter

import os

# before importing pyproj β€” makes coordinate transforms reproducible
os.environ["PROJ_NETWORK"] = "OFF"

# GDAL reads host memory, not a container limit
os.environ["GDAL_CACHEMAX"] = "512"

# do not list a whole directory when opening one file
os.environ["GDAL_DISABLE_READDIR_ON_OPEN"] = "EMPTY_DIR"

These must be set before the libraries are imported, which in a notebook means before the import cell β€” and the kernel must be restarted if they were already imported. Putting them in the environment definition is more reliable:

# pixi.toml
[activation.env]
PROJ_NETWORK = "OFF"
GDAL_CACHEMAX = "512"
GDAL_DISABLE_READDIR_ON_OPEN = "EMPTY_DIR"

PROJ_NETWORK matters more than it looks: with it on, coordinate transformations may fetch grid files at runtime, so the same notebook gives slightly different answers depending on network state. See reprojecting between datums correctly.

A startup module and a memory-managed notebook

The data used on this page

parcels.gpkg is HM Land Registry's INSPIRE Index Polygons for Leeds β€” 293,729 titles, large enough to matter in memory and small enough to load β€” and wards.gpkg is Leeds' 33 wards from the ONS with their Census 2021 population from Nomis. The examples ran in a plain Python process through the same code a notebook cell would; IPython magics ran through IPython. Land Registry data Β© Crown copyright and database right 2026, OGL v3.

from pathlib import Path

import geopandas as gpd
import pandas as pd
import pyogrio
import requests

reply = requests.Session().get("https://use-land-property-data.service.gov.uk/datasets/inspire/download/Leeds.zip",
                               timeout=600)                    # sets a cookie, then redirects to itself
reply.raise_for_status()
Path("leeds_inspire.zip").write_bytes(reply.content)
pyogrio.set_gdal_config_options({"GML_DOWNLOAD_SCHEMA": "NO"})
gpd.read_file("zip://leeds_inspire.zip!Land_Registry_Cadastral_Parcels.gml").to_file("parcels.gpkg")

ONS = "https://services1.arcgis.com/ESMARspQHYMw9BZ9/arcgis/rest/services"
wards = gpd.read_file(f"{ONS}/Wards_December_2024_Boundaries_UK_BFC/FeatureServer/0/query"
                      "?where=LAD24NM%3D%27Leeds%27&outFields=WD24CD,WD24NM&outSR=27700&f=geojson"
                      ).set_crs(27700, allow_override=True)
population = pd.read_csv("https://www.nomisweb.co.uk/api/v01/dataset/NM_2021_1.data.csv?date=latest"
                         f"&geography={','.join(wards['WD24CD'])}&c2021_restype_3=0&measures=20100"
                         "&select=geography_code,obs_value").set_index("GEOGRAPHY_CODE")["OBS_VALUE"]
wards.assign(population=wards["WD24CD"].map(population)).to_file("wards.gpkg")
print(f"parcels.gpkg {Path('parcels.gpkg').stat().st_size / 1e6:.0f} MB, wards.gpkg written")
parcels.gpkg 190 MB, wards.gpkg written

Example 1: a startup module for spatial notebooks

# gis_notebook.py β€” import once at the top of any spatial notebook
"""Sensible defaults for spatial work in Jupyter. Usage:

    from gis_notebook import setup
    setup()
"""
import os
import sys
import warnings
from pathlib import Path

# must precede the library imports
os.environ.setdefault("PROJ_NETWORK", "OFF")
os.environ.setdefault("GDAL_CACHEMAX", "512")
os.environ.setdefault("GDAL_DISABLE_READDIR_ON_OPEN", "EMPTY_DIR")


def setup(*, figsize=(10, 10), retina=True, quiet_warnings=True, verbose=True):
    import matplotlib.pyplot as plt
    import pandas as pd
    import geopandas as gpd

    plt.rcParams.update({
        "figure.figsize": figsize,
        "figure.dpi": 96,
        "savefig.dpi": 200,
        "savefig.bbox": "tight",
        "savefig.facecolor": "white",
        "axes.grid": False,
        "axes.spines.top": False,
        "axes.spines.right": False,
        "font.size": 11,
    })
    pd.set_option("display.max_columns", 50)
    pd.set_option("display.width", 160)
    pd.set_option("display.float_format", lambda v: f"{v:,.4f}")
    gpd.options.display_precision = 4

    if quiet_warnings:
        # keep the CRS warning β€” it is the one that matters
        warnings.filterwarnings("ignore", message=".*initial implementation of Parquet.*")
        warnings.filterwarnings("ignore", category=FutureWarning, module="geopandas")

    try:
        ip = get_ipython()
        ip.run_line_magic("matplotlib", "inline")
        if retina:
            ip.run_line_magic("config", 'InlineBackend.figure_format = "retina"')
        ip.run_line_magic("load_ext", "autoreload")
        ip.run_line_magic("autoreload", "2")
    except (NameError, Exception):
        pass

    if verbose:
        environment()


def environment():
    """Print what is actually running β€” the first thing to check when confused."""
    import geopandas, shapely, pyproj, rasterio, pandas, numpy
    from shapely import geos_version_string
    print(f"python     {sys.version.split()[0]}")
    print(f"executable {sys.executable.replace(str(Path.home()), '~')}")
    print(f"geopandas  {geopandas.__version__}   pandas {pandas.__version__}")
    print(f"shapely    {shapely.__version__}   GEOS {geos_version_string.split('-')[0]}")
    print(f"pyproj     {pyproj.__version__}   PROJ {pyproj.proj_version_str}   "
          f"network {pyproj.network.is_network_enabled()}")
    print(f"rasterio   {rasterio.__version__}   GDAL {rasterio.__gdal_version__}")
    try:
        import psutil
        vm = psutil.virtual_memory()
        print(f"memory     {vm.available / 1e9:.1f} of {vm.total / 1e9:.1f} GB free")
    except ImportError:
        pass
setup()          # in a project: from gis_notebook import setup
python     3.14.4
executable ~/WebstormProjects/spatialworkflow_11ty/.cache/gisenv/bin/python
geopandas  1.1.4   pandas 3.0.6
shapely    2.1.2   GEOS 3.13.1
pyproj     3.8.0   PROJ 9.8.1   network False
rasterio   1.5.1   GDAL 3.12.4
memory     28.6 of 65.4 GB free

Printing the environment on every run is the single highest-value line here. When a notebook behaves differently from yesterday, or from a colleague's, this output answers the question immediately β€” and it is captured in the notebook, so it travels with the analysis.

autoreload 2 re-imports edited modules without restarting the kernel, which is what makes it practical to keep real code in .py files and use the notebook for exploration.

The warning filters are deliberately narrow. Suppressing everything hides the geographic-CRS warning, which is the one warning in this stack that reliably indicates a bug.

Example 2: a notebook that manages its own memory

import gc
import geopandas as gpd

class Workspace:
    """Load layers lazily, report memory, and free them explicitly."""

    def __init__(self, budget_mb=8_000):
        self.layers = {}
        self.budget_mb = budget_mb

    def _rss(self):
        import psutil, os
        return psutil.Process(os.getpid()).memory_info().rss / 1e6

    def load(self, name, path, **kwargs):
        before = self._rss()
        if before > self.budget_mb * 0.8:
            print(f"⚠ already using {before:,.0f} MB of a {self.budget_mb:,} MB "
                  f"budget β€” call .drop() on something first")
        gdf = gpd.read_file(path, **kwargs)
        used = self._rss() - before
        self.layers[name] = gdf
        print(f"{name:<16} {len(gdf):>9,} rows  {used:>7,.0f} MB  "
              f"total {self._rss():,.0f} MB")
        return gdf

    def drop(self, *names):
        for name in names:
            self.layers.pop(name, None)
        gc.collect()
        print(f"freed β€” now {self._rss():,.0f} MB")

    def report(self):
        print(f"{'layer':<16}{'rows':>10}{'MB':>10}  crs")
        for name, gdf in self.layers.items():
            mb = gdf.memory_usage(deep=True).sum() / 1e6
            print(f"{name:<16}{len(gdf):>10,}{mb:>10,.0f}  {gdf.crs}")
        print(f"{'TOTAL (rss)':<16}{'':>10}{self._rss():>10,.0f}")

ws = Workspace(budget_mb=8_000)
ws.load("parcels", "parcels.gpkg", columns=["INSPIREID"])
ws.load("wards", "wards.gpkg")
ws.report()
ws.drop("parcels")
parcels            293,729 rows      300 MB  total 1,576 MB
wards                   33 rows        0 MB  total 1,576 MB
layer                 rows        MB  crs
parcels            293,729         4  EPSG:27700
wards                   33         0  EPSG:27700
TOTAL (rss)                    1,576
freed β€” now 1,556 MB

Notice the gap between what memory_usage reports for the parcels and what loading them actually cost the process: pandas counts the geometry column as one pointer per row, and the Shapely objects behind those pointers β€” the bulk of the memory β€” are invisible to it. That gap is what makes naive capacity planning wrong.

del gdf alone is often not enough in a notebook, because Jupyter keeps references in Out[n] and _, and here the earlier cells still hold gdf. Explicit removal plus gc.collect() is what frees the memory β€” and the resident size may still not fall all the way back, because the allocator keeps freed memory for reuse. Measure; do not assume.

Four answers to how much memory the Leeds parcels take: the file, a rule-of-thumb estimate, what loading them cost, and what pandas reports. 293,729 Leeds parcels, four memory numbersGeoPackage on disk190 MBpeek() estimate, Γ—2.5476 MBmeasured on loading214 MBpandas memory_usage4 MB0250 MB500 MBmemory_usage counts pointers; the geometries live in GEOS, out of its sight spatialworkflow.io
Example 2’s numbers. The estimate was cautious, the measurement is the answer, and pandas misses almost all of it.

Example 3: turning a notebook into something reproducible

# at the top of the notebook, after setup()
from pathlib import Path
import json
from datetime import datetime, timezone
import subprocess

def run_context(inputs=(), out_path="run_context.json"):
    """Capture everything needed to explain this run six months from now."""
    import geopandas, shapely, pyproj, rasterio
    from shapely import geos_version_string

    def git(*args):
        try:
            return subprocess.check_output(["git", *args], text=True,
                                           stderr=subprocess.DEVNULL).strip()
        except Exception:
            return None

    context = {
        "when": datetime.now(timezone.utc).isoformat(),
        "git_commit": git("rev-parse", "HEAD"),
        "git_dirty": bool(git("status", "--porcelain")),
        "versions": {
            "geopandas": geopandas.__version__,
            "shapely": shapely.__version__,
            "geos": geos_version_string.split("-")[0],
            "pyproj": pyproj.__version__,
            "proj": pyproj.proj_version_str,
            "gdal": rasterio.__gdal_version__,
        },
        "proj_network": pyproj.network.is_network_enabled(),
        "inputs": [],
    }
    for path in inputs:
        p = Path(path)
        if p.exists():
            import hashlib
            h = hashlib.sha256()
            with p.open("rb") as f:
                for block in iter(lambda: f.read(1 << 20), b""):
                    h.update(block)
            context["inputs"].append({
                "path": str(p), "bytes": p.stat().st_size,
                "sha256": h.hexdigest()[:16],
                "modified": datetime.fromtimestamp(
                    p.stat().st_mtime, tz=timezone.utc).isoformat(),
            })
    Path(out_path).write_text(json.dumps(context, indent=2))
    print(json.dumps(context, indent=2)[:600])
    return context

ctx = run_context(inputs=["parcels.gpkg", "wards.gpkg"])
{
  "when": "2026-09-27T13:42:17.973733+00:00",
  "git_commit": null,
  "git_dirty": false,
  "versions": {
    "geopandas": "1.1.4",
    "shapely": "2.1.2",
    "geos": "3.13.1",
    "pyproj": "3.8.0",
    "proj": "9.8.1",
    "gdal": "3.12.4"
  },
  "proj_network": false,
  "inputs": [
    {
      "path": "parcels.gpkg",
      "bytes": 190414848,
      "sha256": "b20de3377a7c1fc5",
      "modified": "2026-09-27T13:42:09.438143+00:00"
    },
    {
      "path": "wards.gpkg",
      "bytes": 925696,
      "sha256": "30798e1b00f098ec",
      "modified": "2026-09-27T13:42:11.999542+00:00"
    }
 

Hashing the inputs is what distinguishes this from a version dump. When the numbers change and nothing in the code did, the hash says whether the data changed β€” which is the first question and usually the answer.

git_dirty records whether the working tree had uncommitted changes, so "commit c055798" is not a claim the run cannot support.

Writing this to a file rather than only printing it means the record survives the outputs being stripped, which is exactly when you will want it.

Jupyter's defaults fit small data

Jupyter default assumptions against what spatial work actually does. three assumptions behind the defaultsspatial work doesso you needdata is smallβœ•gigabytesa size check before readingoutput is a line plot!raster mapsbigger figures, nbstripoutstate is cheapβœ•multi-GB framesexplicit drop and collectdel gdf frees nothing while Out[n] still holds a reference spatialworkflow.io
Three assumptions behind the defaults. Spatial work breaks all three.

Jupyter's defaults come from its origins as a teaching and scientific-computing tool, where the data is small, the output is a line plot, and the notebook is the artefact. Spatial work violates all three assumptions, and each of the fixes above corresponds to one of those violations.

The data is large and binary. A GeoDataFrame is not a few thousand floats; it is millions of Python objects wrapping GEOS structures, at two to three times the size of the file on disk. Jupyter has no memory management and no warning mechanism β€” a cell that allocates more than the machine has gets the process killed by the kernel, and Jupyter reports only that the kernel died. Checking the header and estimating before reading is the only prevention, because there is no recovery.

The output is an image. Matplotlib's default 6.4 Γ— 4.8 inches at 100 dpi suits a line chart and is useless for a map, where the information is spatial detail. And because output is stored inside the .ipynb as base64, a notebook with a dozen maps is several megabytes of binary in a JSON file β€” which git treats as one enormous changed line. nbstripout resolves the tension by keeping outputs locally and stripping them on commit.

State is invisible and persistent. A notebook's variables outlive the cells that made them, and cells can run in any order. In ordinary Python work that is a mild hazard; with multi-gigabyte layers it is a memory leak with a user interface. Out[n], _, __ and ___ all hold references, so del gdf frequently frees nothing. Explicit lifecycle management β€” a dict you can drop from, plus gc.collect() β€” is what actually reclaims the memory.

Three notebook symptoms that all come from the kernel not being the interpreter you think. the kernel is a specific interpreter, and it need not be yourswhat you seewhat is actually trueModuleNotFoundError after !pipinstall!pip used PATH’s pip, not thekernel’sThe kernel dies with no messageThe OS killed it; Python never gotto raiseWorks in the shell, not thenotebooksys.executable is a differentPythonPrint sys.executable in the first cell and two of these three stop happening spatialworkflow.io
One environment per project, registered as a named kernel, removes the ambiguity entirely.

The environment is ambiguous. A notebook runs against a kernel, and the kernel is a specific Python interpreter that need not be the one your terminal's pip targets. This single confusion accounts for an enormous share of "I installed it and it still isn't found". Registering a named kernel per project and printing sys.executable at startup removes the ambiguity in two lines.

And the native stack is configured by environment variables read at import. PROJ_NETWORK, GDAL_CACHEMAX and the thread-count variables are all consulted when the C libraries initialise, which in a notebook happens on the first import and never again. Setting them in a cell after importing GeoPandas does nothing, and the failure is silent. Putting them in the environment definition β€” pixi.toml, environment.yml, the kernel spec β€” is the only reliable place.

The wider point is that a notebook is a good place to explore and a poor place to keep logic. The pattern that works is code in .py modules, imported with autoreload so edits take effect immediately, and the notebook holding the narrative, the parameters and the figures. That keeps the reviewable work in files git understands, makes the same code testable and importable by a scheduled job, and leaves the notebook doing what it is genuinely good at β€” showing you a map of what you just computed.

Kernel pip, environment variables and the OOM killer

  • !pip install uses PATH's pip, not the kernel's. Use !{sys.executable} -m pip install.
  • sys.executable in a cell tells you which environment is really running. Check it first, always.
  • Environment variables for GDAL and PROJ are read at import. Set them before, or restart the kernel.
  • A silently dead kernel is almost always the OOM killer. Jupyter cannot report it.
  • del gdf may free nothing β€” Out[n] and _ still hold references. Use %reset -f out or a workspace object.
  • figure_format = "svg" is sharp and becomes slow above a few thousand features.
  • explore() embeds the whole layer as GeoJSON, so sample before calling it on anything large.
  • nbstripout removes interactive maps too, which is correct and surprises collaborators.
  • %%time and %%timeit on a cell are the quickest profiling available; %memit needs memory_profiler.
  • jupyter nbconvert --execute re-runs a notebook top to bottom, which catches hidden state that only worked because of run order.

FAQ

Why does my kernel die with no error message?

Almost always the OOM killer. The process is terminated without warning, so Jupyter can only report that the kernel died. Check the file size and estimate memory before reading.

I installed a package and the notebook still says ModuleNotFoundError.

The kernel is a different environment from the one your terminal's pip targets. Print sys.executable in a cell, and install with !{sys.executable} -m pip install.

How do I make maps bigger?

plt.rcParams["figure.figsize"] = (10, 10) once at the top, plus %config InlineBackend.figure_format = "retina" for a sharper image on a high-DPI screen.

Should I commit notebooks to git?

Yes, with nbstripout installed so outputs are stripped. A notebook with maps is megabytes of base64 in a JSON file, which makes diffs unreadable.

Why is explore() so slow on my data?

It embeds the entire layer as GeoJSON in the notebook output. Sample to a few thousand features first, or use a static plot.

Where should the environment variables go?

In the environment definition β€” pixi.toml, environment.yml, or the kernel spec. Setting them in a cell after importing GeoPandas has no effect, because the C libraries read them at import.

Should my analysis live in the notebook?

Keep logic in .py modules and use the notebook for narrative, parameters and figures. With autoreload 2, edits to the modules take effect immediately, and the code stays testable and reusable by a scheduled job.