How to Add a Progress Bar and ETA to a Long GIS Batch Job
On this page
Problem statement
You start a batch over 3,400 rasters and get this:
$ python batch_clip.pyNothing. For forty minutes. Is it working? Is it stuck on one enormous file? Will it finish before the meeting, or is it going to run all night? The only way to tell is to watch the output folder fill up in another terminal — and even then you cannot tell how long the rest will take.
Long jobs without feedback cost you twice: once in the anxiety of not knowing, and again when you kill a run that was two minutes from finishing, or leave a hung one going for six hours.
What a good progress display has to survive:
- output redirected to a log file, where carriage-return animation becomes garbage
- running under cron, systemd or CI where there is no terminal at all
- parallel workers all wanting to write to the same line
- items of wildly different cost, so "50% of files" is not "50% of the time"
- a need to know not just how far, but how fast and how much longer
Quick answer
Use tqdm when a human is watching, and a periodic log line when one is not:
- detect whether stdout is a terminal —
sys.stdout.isatty() - interactive:
tqdmover the file list, with the current filename in the postfix - non-interactive: log every N items with counts, rate and ETA
- weight the progress by file size when items differ a lot in cost
- always print a final summary with elapsed time and per-item rate
import sys, time
from pathlib import Path
files = sorted(Path("data/raw").glob("*.tif"))
started = time.perf_counter()
if sys.stdout.isatty():
from tqdm import tqdm
iterator = tqdm(files, unit="file", desc="clipping")
else:
iterator = files
for i, path in enumerate(iterator, start=1):
process_one(path)
if not sys.stdout.isatty() and (i % 25 == 0 or i == len(files)):
elapsed = time.perf_counter() - started
rate = i / elapsed
eta = (len(files) - i) / rate
print(f"[{i}/{len(files)}] {i/len(files):5.1%} {rate:5.2f} file/s "
f"eta {eta/60:5.1f} min", flush=True)
print(f"done: {len(files)} files in {(time.perf_counter()-started)/60:.1f} min")[25/76] 32.9% 9.16 file/s eta 0.1 min
[50/76] 65.8% 9.02 file/s eta 0.0 min
[75/76] 98.7% 9.02 file/s eta 0.0 min
[76/76] 100.0% 9.03 file/s eta 0.0 min
done: 76 files in 0.1 min
flush=True matters more than it looks: without it, a redirected stream buffers 4–8 KB before writing, so your log stays empty for the first several minutes.
Choosing the right kind of feedback
Totals first, then bars that survive logs and workers
Know the total before you start
A progress bar needs a denominator, so resolve the work list up front instead of streaming a generator.
from pathlib import Path
import sys
files = sorted(Path("data/raw").rglob("*.tif"))
if not files:
sys.exit("nothing to process")
total_bytes = sum(f.stat().st_size for f in files)
print(f"{len(files)} files, {total_bytes/1e9:.2f} GB to process")76 files, 0.16 GB to process
Reporting the total size as well as the count sets expectations properly when file sizes vary.
Add tqdm for interactive runs
from tqdm import tqdm
with tqdm(total=len(files), unit="file", desc="clipping", dynamic_ncols=True) as bar:
for path in files:
bar.set_postfix_str(path.name[:32], refresh=False)
process_one(path)
bar.update(1)tqdm gives you percentage, elapsed, ETA and rate with no arithmetic on your part. Two habits keep it readable: put the current filename in the postfix rather than the description, and use dynamic_ncols=True so it adapts to the terminal width.
To weight by cost rather than count, drive the bar in bytes:
with tqdm(total=total_bytes, unit="B", unit_scale=True, desc="clipping") as bar:
for path in files:
process_one(path)
bar.update(path.stat().st_size)Now a 4 GB raster moves the bar four thousand times further than a 1 MB one, and the ETA stops lying.
Make it behave when nobody is watching
A carriage-return bar written to a file produces a single line thousands of characters long. Detect the context and switch.
import sys, logging, time
log = logging.getLogger("batch")
class Progress:
"""A progress reporter that works in a terminal, a log file, or CI."""
def __init__(self, total, every=25, desc="working"):
self.total, self.every, self.desc = total, every, desc
self.n, self.started = 0, time.perf_counter()
self.tty = sys.stdout.isatty()
self.bar = None
if self.tty:
try:
from tqdm import tqdm
self.bar = tqdm(total=total, unit="item", desc=desc, dynamic_ncols=True)
except ImportError:
self.bar = None
def update(self, n=1, item=""):
self.n += n
if self.bar is not None:
if item:
self.bar.set_postfix_str(item[:32], refresh=False)
self.bar.update(n)
elif self.n % self.every == 0 or self.n == self.total:
log.info("%s", self.line())
def line(self) -> str:
elapsed = time.perf_counter() - self.started
rate = self.n / elapsed if elapsed else 0
eta = (self.total - self.n) / rate if rate else float("inf")
return (f"[{self.n}/{self.total}] {self.n/self.total:5.1%} "
f"{rate:6.2f}/s elapsed {elapsed/60:5.1f}m eta {eta/60:5.1f}m")
def close(self):
if self.bar is not None:
self.bar.close()
elapsed = time.perf_counter() - self.started
log.info("finished %d items in %.1f min (%.2f/s)",
self.n, elapsed / 60, self.n / elapsed if elapsed else 0)logging.basicConfig(level=logging.INFO, format="%(message)s", stream=sys.stdout)
progress = Progress(len(files), desc="reprojecting")
for path in files:
process_one(path)
progress.update(item=path.name)
progress.close()[25/76] 32.9% 9.76/s elapsed 0.0m eta 0.1m
[50/76] 65.8% 9.53/s elapsed 0.1m eta 0.0m
[75/76] 98.7% 9.57/s elapsed 0.1m eta 0.0m
[76/76] 100.0% 9.59/s elapsed 0.1m eta 0.0m
finished 76 items in 0.1 min (9.59/s)
One class, and the same script is pleasant interactively and readable in journalctl.
Report progress from parallel workers
Workers cannot share a terminal line, but the parent can count completions.
from concurrent.futures import ProcessPoolExecutor, as_completed
from tqdm import tqdm
failed = []
with ProcessPoolExecutor(max_workers=4) as pool:
futures = {pool.submit(process_one, str(p)): p for p in files}
with tqdm(total=len(futures), unit="file", desc="clipping") as bar:
for fut in as_completed(futures):
path = futures[fut]
try:
fut.result()
except Exception as exc:
failed.append(path)
if len(failed) <= 2:
bar.write(f"failed {path.name}: {exc}") # never use print() with tqdm
bar.set_postfix_str(path.name[:28], refresh=False)
bar.update(1)
print(f"{len(files) - len(failed)} ok, {len(failed)} failed")failed Copernicus_DSM_COG_30_N49_00_E000_00_DEM.tif: A process in the process pool was terminated abruptly while the future was running or pending.
failed Copernicus_DSM_COG_30_N49_00_W001_00_DEM.tif: A process in the process pool was terminated abruptly while the future was running or pending.
0 ok, 76 failed
Every file failed, and the bar still ran to 100% in a fraction of a second — progress counts completions, not successes. The cause is Python 3.14's default start method on Linux, forkserver: a worker starts fresh and imports the function it is sent by module name, and process_one was defined in a notebook, which has no importable module. (fork, the old default, copied the parent's memory and hid this.) Put the worker function in a module:
%%writefile tasks.py
from pathlib import Path
import rasterio
from rasterio.warp import Resampling, calculate_default_transform, reproject
def process_one(path, out=Path("data/out")):
"""Reproject one DEM tile to the British National Grid at 90 m."""
path = Path(path)
with rasterio.open(path) as src:
transform, width, height = calculate_default_transform(
src.crs, "EPSG:27700", src.width, src.height, *src.bounds, resolution=90)
profile = src.profile | {"driver": "GTiff", "crs": "EPSG:27700", "transform": transform,
"width": width, "height": height, "compress": "deflate"}
with rasterio.open(out / path.name, "w", **profile) as dst:
reproject(rasterio.band(src, 1), rasterio.band(dst, 1), resampling=Resampling.bilinear)import time
import tasks
started, failed = time.perf_counter(), []
with ProcessPoolExecutor(max_workers=4) as pool:
futures = {pool.submit(tasks.process_one, str(p)): p for p in files}
for fut in as_completed(futures):
if fut.exception():
failed.append(futures[fut])
print(f"{len(files) - len(failed)} ok, {len(failed)} failed in {time.perf_counter() - started:.1f} s")76 ok, 0 failed in 2.4 s
bar.write() prints above the bar without corrupting it. A plain print() inside a tqdm loop is the classic way to end up with a smeared terminal.
Estimate remaining time honestly
The naive ETA — remaining items divided by the average rate — is badly wrong when items differ in cost or the machine warms up. Two cheap improvements:
import time
from collections import deque
class Eta:
"""ETA from a moving window, optionally weighted by item size."""
def __init__(self, total_units, window=50):
self.total, self.done = total_units, 0
self.samples = deque(maxlen=window)
self.last = time.perf_counter()
def tick(self, units=1):
now = time.perf_counter()
self.samples.append((units, now - self.last))
self.last = now
self.done += units
def seconds_left(self) -> float:
units = sum(u for u, _ in self.samples)
secs = sum(t for _, t in self.samples)
if not units or not secs:
return float("inf")
return (self.total - self.done) / (units / secs)Whether weighting by size is worth it depends on where the time goes, and that can be measured. Time every tile, then ask what each method would have predicted a quarter, half and three quarters of the way through:
import numpy as np
times, sizes = [], []
for path in files:
t0 = time.perf_counter()
process_one(path)
times.append(time.perf_counter() - t0)
sizes.append(path.stat().st_size)
times, sizes = np.array(times), np.array(sizes)
print(f"per tile: {times.min():.3f}–{times.max():.3f} s for {sizes.min() / 1e6:.2f}–{sizes.max() / 1e6:.2f} MB; "
f"correlation {np.corrcoef(sizes, times)[0, 1]:.2f}")
for frac in (0.25, 0.5, 0.75):
k = int(len(files) * frac)
elapsed, actual = times[:k].sum(), times[k:].sum()
by_count = elapsed / k * (len(files) - k)
by_bytes = elapsed / sizes[:k].sum() * sizes[k:].sum()
print(f"at {frac:.0%}: {actual:4.1f} s left; count ETA {by_count:4.1f} s, byte ETA {by_bytes:4.1f} s")per tile: 0.088–0.136 s for 0.05–4.22 MB; correlation 0.85
at 25%: 6.0 s left; count ETA 5.8 s, byte ETA 6.7 s
at 50%: 4.0 s left; count ETA 4.0 s, byte ETA 3.6 s
at 75%: 2.0 s left; count ETA 2.0 s, byte ETA 1.5 s
Time and size are correlated here, and the byte-weighted ETA is still no better than counting — worse, late in the run. Every tile is a 1° square with the same number of cells, so most of the cost is fixed per tile, and the compressed size — which runs from almost nothing for a tile of sea to 4 MB for the Highlands — mostly measures how much land it holds. Weighting by bytes over-predicts after a run of small tiles and under-predicts after large ones. Bytes are the right weight when the work really is proportional to the data read — parsing vector features, say — and the wrong one when a fixed cost dominates. Measure before choosing.
A moving window tracks the current rate rather than the lifetime average, so an ETA recovers quickly after a slow patch. Weighting by bytes handles the "one 4 GB file among a thousand small ones" case, which no count-based estimate can.
Make the summary the important part
import time
started, ok, failed = time.perf_counter(), [], []
for path in files:
try:
process_one(path)
ok.append(path)
except Exception:
failed.append(path)
summary = {"total": len(files), "ok": len(ok), "failed": len(failed),
"elapsed_s": round(time.perf_counter() - started, 1)}
summary["rate_per_min"] = round(summary["total"] / (summary["elapsed_s"] / 60), 1)
print("\n─── summary ───")
for k, v in summary.items():
print(f"{k:14} {v}")
─── summary ───
total 76
ok 76
failed 0
elapsed_s 8.1
rate_per_min 563.0
The bar is for the person watching; the summary is what ends up in the log, the ticket and next week's capacity estimate.
Tracking 76 Copernicus DEM tiles
The data used on this page
A real batch: the 76 tiles of the Copernicus GLO-90 DEM (© DLR e.V. 2010–2014 and © Airbus Defence and Space GmbH 2014–2018, provided under COPERNICUS by the European Union and ESA) that cover Great Britain, Ireland and the Channel coast, one 1° × 1° GeoTIFF each, in data/raw/. Coastal tiles are mostly sea and compress to almost nothing, so the files range from 0.05 to 4.2 MB. process_one is the job: reproject a tile to the British National Grid at 90 m.
from pathlib import Path
import rasterio
import requests
from rasterio.warp import Resampling, calculate_default_transform, reproject
raw, out = Path("data/raw"), Path("data/out")
raw.mkdir(parents=True, exist_ok=True)
out.mkdir(parents=True, exist_ok=True)
BUCKET = "https://copernicus-dem-90m.s3.amazonaws.com"
names = requests.get(f"{BUCKET}/tileList.txt", timeout=60).text.split()
wanted = {f"N{lat:02d}_00_{ew}{lon:03d}_00" for lat in range(49, 59)
for ew, lon in [("W", w) for w in range(1, 9)] + [("E", 0)]}
wanted |= {f"N{lat:02d}_00_E001_00" for lat in range(50, 54)}
tiles = sorted(n for n in names if n[22:36] in wanted)
for name in tiles:
path = raw / f"{name}.tif"
if not path.exists():
reply = requests.get(f"{BUCKET}/{name}/{name}.tif", timeout=120)
reply.raise_for_status()
path.write_bytes(reply.content)
def process_one(path):
"""Reproject one DEM tile to the British National Grid at 90 m."""
path = Path(path)
with rasterio.open(path) as src:
transform, width, height = calculate_default_transform(
src.crs, "EPSG:27700", src.width, src.height, *src.bounds, resolution=90)
profile = src.profile | {"driver": "GTiff", "crs": "EPSG:27700", "transform": transform,
"width": width, "height": height, "compress": "deflate"}
with rasterio.open(out / path.name, "w", **profile) as dst:
reproject(rasterio.band(src, 1), rasterio.band(dst, 1), resampling=Resampling.bilinear)
sizes = sorted(p.stat().st_size for p in raw.glob("*.tif"))
print(f"{len(tiles)} tiles, {sizes[0] / 1e6:.2f}–{sizes[-1] / 1e6:.2f} MB each")76 tiles, 0.05–4.22 MB each
Example 1: a complete batch with size-weighted progress
%%writefile batch_reproject.py
from pathlib import Path
import logging, sys, time
import tasks
logging.basicConfig(level=logging.INFO, format="%(levelname)s %(message)s", stream=sys.stdout)
log = logging.getLogger("batch")
SRC, OUT = Path("data/raw"), Path("data/out")
def process_one(path: Path) -> None:
tasks.process_one(path, OUT)
def main() -> int:
files = sorted(SRC.rglob("*.tif"))
if not files:
log.error("nothing to process in %s", SRC)
return 2
OUT.mkdir(parents=True, exist_ok=True)
total_bytes = sum(f.stat().st_size for f in files)
log.info("%d files, %.2f GB", len(files), total_bytes / 1e9)
tty = sys.stdout.isatty()
bar = None
if tty:
from tqdm import tqdm
bar = tqdm(total=total_bytes, unit="B", unit_scale=True, desc="converting")
started, done_bytes, ok, failed = time.perf_counter(), 0, 0, []
for i, path in enumerate(files, start=1):
size = path.stat().st_size
try:
process_one(path)
ok += 1
except Exception as exc:
failed.append((path.name, f"{type(exc).__name__}: {exc}"))
(bar.write if bar else log.warning)(f"failed {path.name}: {exc}")
done_bytes += size
if bar:
bar.set_postfix_str(path.name[:28], refresh=False)
bar.update(size)
elif i % 25 == 0 or i == len(files):
elapsed = time.perf_counter() - started
frac = done_bytes / total_bytes
eta = elapsed / frac - elapsed if frac else float("inf")
log.info("[%d/%d] %5.1f%% eta %.1f min", i, len(files), frac * 100, eta / 60)
if bar:
bar.close()
elapsed = time.perf_counter() - started
log.info("%d ok, %d failed in %.1f min", ok, len(failed), elapsed / 60)
for name, err in failed:
log.warning(" ! %s: %s", name, err)
return 1 if failed else 0
if __name__ == "__main__":
raise SystemExit(main())!{sys.executable} batch_reproject.py > run.log
print(Path("run.log").read_text())INFO 76 files, 0.16 GB
INFO [25/76] 29.6% eta 0.1 min
INFO [50/76] 69.8% eta 0.0 min
INFO [75/76] 99.9% eta 0.0 min
INFO [76/76] 100.0% eta 0.0 min
INFO 76 ok, 0 failed in 0.2 min
Example 2: nested bars for a two-level job
from itertools import groupby
from tqdm import tqdm
rows = [(band, list(tiles)) for band, tiles in groupby(files, key=lambda p: p.name[22:25])] # N49 … N58
outer = tqdm(rows, unit="row", desc="latitude", position=0)
for band, tiles in outer:
outer.set_postfix_str(band)
for path in tqdm(tiles, unit="file", desc=band, position=1, leave=False):
process_one(path)
print(len(rows), "rows of tiles,", sum(len(t) for _, t in rows), "tiles")10 rows of tiles, 76 tiles
leave=False on the inner bar keeps the display to two lines instead of one per folder.
Example 3: progress for a single long operation
Some steps are one call that takes twenty minutes. Report from inside it.
import rasterio
from rasterio.windows import Window
from tqdm import tqdm
dem = "data/raw/Copernicus_DSM_COG_30_N54_00_W004_00_DEM.tif" # the Lake District
with rasterio.open(dem) as src:
print(f"{src.width} × {src.height} cells, internal blocks {src.block_shapes[0]}: "
f"{len(list(src.block_windows(1)))} block window")
# one 2048 × 2048 block covers the whole tile, so iterate strips of rows instead
windows = [Window(0, row, src.width, min(100, src.height - row)) for row in range(0, src.height, 100)]
profile = src.profile
with rasterio.open("data/out/dem_ft.tif", "w", **profile) as dst:
for window in tqdm(windows, unit="strip", desc="rescaling"):
dst.write((src.read(1, window=window) * 3.28084).astype(profile["dtype"]),
1, window=window)
print(len(windows), "strips")800 × 1200 cells, internal blocks (2048, 2048): 1 block window
12 strips
For GDAL and Processing operations, a callback gives you the same thing:
from osgeo import gdal
from tqdm import tqdm
gdal.UseExceptions()
bar = tqdm(total=100, unit="%", desc="warping")
def gdal_progress(complete, message, _):
bar.n = int(complete * 100)
bar.refresh()
return 1
gdal.Warp("data/out/warped.tif", dem, dstSRS="EPSG:3857", callback=gdal_progress)
bar.close()
print("callback reached", bar.n, "%")callback reached 100 %
Example 4: machine-readable progress for a dashboard
import json, time
from pathlib import Path
STATE = Path("logs/progress.json")
def publish(done, total, started, current=""):
elapsed = time.perf_counter() - started
rate = done / elapsed if elapsed else 0
STATE.parent.mkdir(parents=True, exist_ok=True)
STATE.write_text(json.dumps({
"done": done, "total": total, "pct": round(done / total * 100, 1),
"rate_per_s": round(rate, 3),
"eta_s": round((total - done) / rate) if rate else None,
"current": current, "updated": time.time(),
}), encoding="utf-8")publish(40, len(files), time.perf_counter() - 12.5, current=files[39].name)
print(STATE.read_text()){"done": 40, "total": 76, "pct": 52.6, "rate_per_s": 3.2, "eta_s": 11, "current": "Copernicus_DSM_COG_30_N53_00_W006_00_DEM.tif", "updated": 1790712821.6584823}
A JSON file that another process can poll turns "is it still going?" into a query rather than a guess — and its updated timestamp doubles as a liveness check.
What progress reporting is for
Progress reporting looks cosmetic and is really an operational feature. Its job is to answer three questions cheaply: is the job alive, how far has it got, and when will it finish. Each of those changes a decision — whether to wait, whether to kill, whether to start something else — so an hour spent on it pays for itself the first time you avoid killing a run that was nearly done.
The two contexts a batch runs in want opposite things. A terminal can redraw one line, so an animated bar is ideal: it is compact, continuously updated, and disappears when the job ends. A log file cannot — carriage returns produce one monstrous line, and the timestamps that make a log useful are missing. So the switch on sys.stdout.isatty() is not a nicety; it is the difference between a readable journalctl and an unreadable one.
The ETA deserves more thought than it usually gets. Remaining items divided by average rate assumes every item costs the same, which is often untrue for GIS data: file sizes span orders of magnitude, and geometry complexity varies more than that. Weighting progress by bytes is the cheapest correction — when the work scales with the data read. When a fixed cost per item dominates, as it does for the equal-sized DEM tiles on this page, bytes mislead and a plain count wins; the measurement beside the Eta class takes a few lines. A moving window over recent items handles the other distortion — a machine that was slow while a cache warmed, or a run that has just hit a large batch of small files.
Parallel jobs need a different structure again. Workers are separate processes with their own stdout, so they must not draw the bar. The parent, which already collects futures as they complete, is the natural place to count, and tqdm.write() exists precisely so error messages can be printed without corrupting the display.
And whichever style you use, the closing summary is the part that persists. The bar is gone the moment the job ends; the summary line — items, failures, elapsed, rate — is what tells you next month whether the job is getting slower.
Buffering, stray prints and stderr
flush=Trueor nothing appears: Redirected stdout is block-buffered. Set it per print, or run Python with-u/PYTHONUNBUFFERED=1.- Never
print()inside atqdmloop: Usebar.write(), or the bar and your message will overwrite each other. tqdmwrites to stderr by default: Handy — it keeps stdout clean for real output. Passfile=sys.stdoutif your logging setup expects otherwise.- A generator has no length:
tqdmcannot show a percentage withouttotal=. Materialise the list, or pass the count you already know. - CI logs hate animation: GitHub Actions renders every bar update as a new line. Detect CI (
os.environ.get("CI")) and fall back to periodic lines. tqdmcosts almost nothing: Roughly microseconds per update, but avoid updating it inside a tight inner loop over millions of geometries; update per file instead.- An ETA is an estimate: State it as one. Reporting "eta 12.4 min" to two decimals implies a precision that does not exist.
Internal links
- How to Batch Process a Folder of GIS Files in Python: The Complete Workflow
- How to Log and Summarise Errors in a Batch GIS Job in Python
- How to Build a Resumable Batch GIS Job in Python
- How to Speed Up Batch GIS Jobs with Parallel Processing in Python
- Batch Script Stops at the First Bad File: How to Keep It Running
- My GIS Pipeline Fails Silently: Fixing Swallowed Errors and Wrong Exit Codes
FAQ
Why does my progress bar look broken in a log file?
Because it redraws using carriage returns, which a file records literally. Detect sys.stdout.isatty() and switch to periodic log lines when there is no terminal.
How do I show progress from parallel workers?
Count completions in the parent. Submit futures, iterate as_completed(), and update one bar there — workers should never draw to the shared terminal.
Why is my ETA so inaccurate?
It is probably counting files, not work. Weight progress by file size and compute the rate from a moving window of recent items rather than the whole run.
Do I have to use tqdm?
No. A print every N items with count, percentage and elapsed time covers most needs and has no dependency. tqdm is simply the nicest interactive option.
Why does nothing appear until the job finishes?
Output buffering. Pass flush=True to print, run with python -u, or set PYTHONUNBUFFERED=1 in the environment.
Can I show progress inside a single long GDAL or Processing call?
Yes — both support progress callbacks. gdal.Warp(..., callback=fn) and QgsProcessingFeedback.setProgress() let you drive a bar from inside an operation you did not write.
How do I let something else monitor the job?
Write a small JSON state file every few seconds with done, total, rate and a timestamp. Another process can poll it, and a stale timestamp doubles as a hung-job alert.