🔧 Fix Privacy & sensitive locations Files & formats Geoprivacy & Anonymisation · step 20 of 20 Updated

A shared file still contains the original coordinates

Problem statement

The points were masked, the file was written, the zip went out — and the true coordinates went with it. Not in the geometry column, which was masked correctly, but in one of the places nobody looks: a second layer in the same GeoPackage, an index column that preserves the source row order, an attribute holding a postcode, a photograph's EXIF block, or a vector map export where every marker carries its own exact position.

These are not exotic attacks. They are the ordinary behaviour of the tools. A GeoPackage is a database and keeps what you put in it. A GeoJSON written by GeoPandas carries fifteen decimal places. A matplotlib SVG of a deliberately blurred scatter plot contained 3,193 <use> elements, one per plotted marker plus the axis ticks, each with its own coordinates.

Quick answer

Read the artefact back and check the containers, not the DataFrame:

import fiona, json, pathlib, piexif, re

# 1. layers in the container
assert fiona.listlayers("release.gpkg") == ["cases_masked"], fiona.listlayers("release.gpkg")

# 2. columns that should not be there
banned = re.compile(r"postcode|uprn|address|easting|northing|lat|lon|index|fid", re.I)
assert not [c for c in released.columns if banned.search(c)]

# 3. coordinate precision actually written
coords = json.loads(pathlib.Path("release.geojson").read_text())["features"][0]["geometry"]["coordinates"]
assert max(len(str(c).split(".")[1]) for c in coords) <= 5, coords

# 4. photographs
assert not [p for p in pathlib.Path("photos").glob("**/*.jpg") if piexif.load(str(p))["GPS"]]

Each of these is a real failure that has shipped in real releases.

Triage of six places original coordinates survive a masked release and the fix for each.
None of these are in the geometry column, which is the only place anyone checks.

Step-by-step solution

1. List the layers in the container

A GeoPackage holds many layers, and to_file(..., layer="masked") on a file that already contains cases adds a layer rather than replacing the file. The same applies to a FileGDB, a SpatiaLite database and a multi-sheet spreadsheet.

2. Drop index and identifier columns

reset_index() leaves an index column. to_file writes an fid. A source_id that joins to the internal system is a key to everything. Drop them explicitly and assert they are gone.

3. Shuffle the rows

Row order is a join key. If the masked file is written in the source's order, row i is row i and any copy of the source restores every true point exactly.

released = released.sample(frac=1, random_state=SEED).reset_index(drop=True)

4. Check the coordinate precision that was written

GeoPandas writes doubles. A masked GeoJSON came out with coordinates like -0.175253311870371 — fifteen decimal places — and at 406 KB. With COORDINATE_PRECISION=5 the same file was 346 KB. That is a bandwidth saving, not a privacy control, but full precision in a file whose stated accuracy is 300 m is a signal that nothing was truncated anywhere.

5. Scan the attributes for coordinates in disguise

A postcode column, an easting/northing pair, a what3words string, a URL containing a latitude, or a free-text note saying "opposite the church" are all coordinates. Regex the column names and the values.

6. Check the attached photographs

EXIF survives copying, zipping and most resizing. Strip it, then verify — How to strip GPS coordinates from photo metadata in Python.

7. Check the vector exports

An SVG or PDF map contains geometry, not pixels. A scatter drawn at 5% opacity looks like a blur and is a list of exact positions. Export rasters (PNG at a stated resolution) for anything derived from sensitive points, or plot the masked points only.

8. Check what the zip actually contains

Packaging is where stale files reappear: a _backup folder, an editor swap file, a .shp from last week, a notebook with outputs. List the archive before sending it.

Stack of a delivery zip showing geopackage layers, geojson precision, photographs, exports and stale files.
Six containers in one delivery, each with its own way of keeping what you removed.

Code examples

Example 1 — prove the vector export carries the points 

import io, re
import matplotlib
matplotlib.use("Agg")
import matplotlib.pyplot as plt

fig, ax = plt.subplots(figsize=(4, 3))
ax.scatter(xy[:, 0], xy[:, 1], s=1, alpha=0.05)     # looks like a soft blur
buf = io.StringIO(); fig.savefig(buf, format="svg"); plt.close(fig)
svg = buf.getvalue()

print(f"SVG {len(svg)/1024:.0f} KB, marker elements: {len(re.findall(r'<use ', svg)):,} "
      f"(points plotted: {len(xy):,})")
SVG 466 KB, marker elements: 3,193 (points plotted: 3,109)

The count exceeds the point count because the axis ticks are drawn with the same element. The blur exists only in the renderer; the coordinates are in the file.

Example 2 — a container audit 

import fiona, pathlib, zipfile, re

def audit_delivery(path):
    path = pathlib.Path(path)
    findings = []

    for gpkg in path.glob("**/*.gpkg"):
        layers = fiona.listlayers(str(gpkg))
        if len(layers) > 1:
            findings.append(f"{gpkg.name}: {len(layers)} layers {layers}")

    for geojson in path.glob("**/*.geojson"):
        text = geojson.read_text()
        m = re.search(r"\[\s*(-?\d+\.(\d+))", text)
        if m and len(m.group(2)) > 7:
            findings.append(f"{geojson.name}: {len(m.group(2))} decimal places written")

    for svg in list(path.glob("**/*.svg")) + list(path.glob("**/*.pdf")):
        findings.append(f"{svg.name}: vector export — confirm it was built from masked points")

    for junk in path.glob("**/*"):
        if junk.suffix in {".bak", ".swp", ".tmp"} or junk.name.startswith("~"):
            findings.append(f"{junk.name}: stale file in the delivery")

    return findings

for f in audit_delivery("delivery/"):
    print(" !", f)

Example 3 — write the release deliberately 

import geopandas as gpd, pathlib

out = pathlib.Path("release.gpkg")
if out.exists():
    out.unlink()                                     # do not append to an existing container

clean = (masked
         .drop(columns=[c for c in masked.columns
                        if c.lower() in {"index", "fid", "objectid", "source_id",
                                         "postcode", "easting", "northing"}])
         .sample(frac=1, random_state=20260915)
         .reset_index(drop=True))

clean.to_file(out, layer="cases_masked", driver="GPKG")
clean.to_file("release.geojson", driver="GeoJSON", COORDINATE_PRECISION=5)

assert fiona.listlayers(out) == ["cases_masked"]

Deleting the target before writing is the single most effective line here. Appending to an existing GeoPackage is the default behaviour and the commonest way a source layer ships.

Explanation

Why the geometry column is the only thing anyone checks

It is the thing that was changed, so it is the thing that gets reviewed. Everything else in the delivery was not changed, which is exactly why it still holds what it held before. The review has to be of the artefact, not of the diff.

Why row order is a complete leak

Masking is applied row-wise and preserves order. If the source is available — an earlier release, an internal file, a copy held by a partner — joining by position recovers every original coordinate with no cleverness at all. Shuffling costs one line and closes it.

Why vector exports are so easy to miss

A PNG is pixels and cannot be inverted beyond its resolution. An SVG or PDF is a scene graph with one element per feature. The visual similarity between the two is total, and the difference in disclosure is complete. Anything intended as a picture should be rasterised at a stated resolution.

Why the GeoPackage default surprises people

GeoPandas' to_file on a GeoPackage adds or replaces a layer within an existing file rather than replacing the file. That is correct database behaviour, and it means a file that has been written to twice contains both states. Delete and rewrite, or use an explicit fresh path.

Two panels comparing a raster PNG export with an SVG export of the same transparent scatter, the SVG carrying one marker element with exact coordinates per point.
The two look identical on screen and differ completely in what they disclose.

Edge cases or notes

  • Shapefiles come in pieces. Shipping the .dbf from a different version than the .shp mixes two releases.
  • Notebook outputs are data. A committed .ipynb can contain a printed head of the unmasked frame.
  • Parquet keeps column statistics. Min/max per row group are stored in the footer for every column present.
  • .qgz projects embed queries and sometimes data. Check before shipping a project file.
  • Editor artefacts travel. .swp, ~ and .bak files are frequently zipped by accident.
  • Cloud storage keeps versions. Overwriting a bucket object does not delete the earlier one.
  • Email attachments are forever. A file sent once cannot be recalled by fixing the pipeline.
  • The fix is a gate, not a habit. Automate the audit; a checklist a person runs is a checklist a person skips.

FAQ

Where do original coordinates hide in a masked release?

In a second layer of the same GeoPackage, in an index column that preserves row order, in attributes such as postcode or easting/northing, in photograph EXIF, in vector map exports, and in stale files inside the delivery zip.

Why does my GeoPackage have two layers?

Because to_file adds a layer to an existing container. Delete the file before writing, or write to a fresh path.

How does row order leak coordinates?

The masked file keeps the source's order, so row i of the release is row i of the source. Anyone with a copy of the source recovers every true point by position. Shuffle before writing.

Is an SVG of a heat map safe to publish?

No. The file contains one element per point with its exact position — 3,193 marker elements in a test export of 3,109 points. Rasterise instead.

Does reducing coordinate precision on write protect anyone?

No. Five decimal places is under a metre. It is a file-size control; the privacy control is the mask or the grid.

How do I stop this happening again?

Put the audit in the publishing pipeline so it reads the built artefact and raises, rather than relying on a review step that a person performs by hand.