How to Add a Progress Bar and ETA to a Long GIS Batch Job
Problem statement
You start a batch over 3,400 rasters and get this:
$ python batch_clip.pyNothing. For forty minutes. Is it working? Is it stuck on one enormous file? Will it finish before the meeting, or is it going to run all night? The only way to tell is to watch the output folder fill up in another terminal — and even then you cannot tell how long the rest will take.
Long jobs without feedback cost you twice: once in the anxiety of not knowing, and again when you kill a run that was two minutes from finishing, or leave a hung one going for six hours.
What a good progress display has to survive:
- output redirected to a log file, where carriage-return animation becomes garbage
- running under cron, systemd or CI where there is no terminal at all
- parallel workers all wanting to write to the same line
- items of wildly different cost, so "50% of files" is not "50% of the time"
- a need to know not just how far, but how fast and how much longer
Quick answer
Use tqdm when a human is watching, and a periodic log line when one is not:
- detect whether stdout is a terminal —
sys.stdout.isatty() - interactive:
tqdmover the file list, with the current filename in the postfix - non-interactive: log every N items with counts, rate and ETA
- weight the progress by file size when items differ a lot in cost
- always print a final summary with elapsed time and per-item rate
import sys, time
from pathlib import Path
files = sorted(Path("data/raw").glob("*.tif"))
started = time.perf_counter()
if sys.stdout.isatty():
from tqdm import tqdm
iterator = tqdm(files, unit="file", desc="clipping")
else:
iterator = files
for i, path in enumerate(iterator, start=1):
process_one(path)
if not sys.stdout.isatty() and (i % 25 == 0 or i == len(files)):
elapsed = time.perf_counter() - started
rate = i / elapsed
eta = (len(files) - i) / rate
print(f"[{i}/{len(files)}] {i/len(files):5.1%} {rate:5.2f} file/s "
f"eta {eta/60:5.1f} min", flush=True)
print(f"done: {len(files)} files in {(time.perf_counter()-started)/60:.1f} min")flush=True matters more than it looks: without it, a redirected stream buffers 4–8 KB before writing, so your log stays empty for the first several minutes.
Choosing the right kind of feedback
Step-by-step solution
Know the total before you start
A progress bar needs a denominator, so resolve the work list up front instead of streaming a generator.
from pathlib import Path
import sys
files = sorted(Path("data/raw").rglob("*.tif"))
if not files:
sys.exit("nothing to process")
total_bytes = sum(f.stat().st_size for f in files)
print(f"{len(files)} files, {total_bytes/1e9:.2f} GB to process")Reporting the total size as well as the count sets expectations properly when file sizes vary.
Add tqdm for interactive runs
from tqdm import tqdm
with tqdm(total=len(files), unit="file", desc="clipping", dynamic_ncols=True) as bar:
for path in files:
bar.set_postfix_str(path.name[:32], refresh=False)
process_one(path)
bar.update(1)tqdm gives you percentage, elapsed, ETA and rate with no arithmetic on your part. Two habits keep it readable: put the current filename in the postfix rather than the description, and use dynamic_ncols=True so it adapts to the terminal width.
To weight by cost rather than count, drive the bar in bytes:
with tqdm(total=total_bytes, unit="B", unit_scale=True, desc="clipping") as bar:
for path in files:
process_one(path)
bar.update(path.stat().st_size)Now a 4 GB raster moves the bar four thousand times further than a 1 MB one, and the ETA stops lying.
Make it behave when nobody is watching
A carriage-return bar written to a file produces a single line thousands of characters long. Detect the context and switch.
import sys, logging, time
log = logging.getLogger("batch")
class Progress:
"""A progress reporter that works in a terminal, a log file, or CI."""
def __init__(self, total, every=25, desc="working"):
self.total, self.every, self.desc = total, every, desc
self.n, self.started = 0, time.perf_counter()
self.tty = sys.stdout.isatty()
self.bar = None
if self.tty:
try:
from tqdm import tqdm
self.bar = tqdm(total=total, unit="item", desc=desc, dynamic_ncols=True)
except ImportError:
self.bar = None
def update(self, n=1, item=""):
self.n += n
if self.bar is not None:
if item:
self.bar.set_postfix_str(item[:32], refresh=False)
self.bar.update(n)
elif self.n % self.every == 0 or self.n == self.total:
log.info("%s", self.line())
def line(self) -> str:
elapsed = time.perf_counter() - self.started
rate = self.n / elapsed if elapsed else 0
eta = (self.total - self.n) / rate if rate else float("inf")
return (f"[{self.n}/{self.total}] {self.n/self.total:5.1%} "
f"{rate:6.2f}/s elapsed {elapsed/60:5.1f}m eta {eta/60:5.1f}m")
def close(self):
if self.bar is not None:
self.bar.close()
elapsed = time.perf_counter() - self.started
log.info("finished %d items in %.1f min (%.2f/s)",
self.n, elapsed / 60, self.n / elapsed if elapsed else 0)progress = Progress(len(files), desc="clipping")
for path in files:
process_one(path)
progress.update(item=path.name)
progress.close()One class, and the same script is pleasant interactively and readable in journalctl.
Report progress from parallel workers
Workers cannot share a terminal line, but the parent can count completions.
from concurrent.futures import ProcessPoolExecutor, as_completed
from tqdm import tqdm
with ProcessPoolExecutor(max_workers=4) as pool:
futures = {pool.submit(process_one, str(p)): p for p in files}
with tqdm(total=len(futures), unit="file", desc="clipping") as bar:
for fut in as_completed(futures):
path = futures[fut]
try:
fut.result()
except Exception as exc:
bar.write(f"failed {path.name}: {exc}") # never use print() with tqdm
bar.set_postfix_str(path.name[:28], refresh=False)
bar.update(1)bar.write() prints above the bar without corrupting it. A plain print() inside a tqdm loop is the classic way to end up with a smeared terminal.
Estimate remaining time honestly
The naive ETA — remaining items divided by the average rate — is badly wrong when items differ in cost or the machine warms up. Two cheap improvements:
import time
from collections import deque
class Eta:
"""ETA from a moving window, optionally weighted by item size."""
def __init__(self, total_units, window=50):
self.total, self.done = total_units, 0
self.samples = deque(maxlen=window)
self.last = time.perf_counter()
def tick(self, units=1):
now = time.perf_counter()
self.samples.append((units, now - self.last))
self.last = now
self.done += units
def seconds_left(self) -> float:
units = sum(u for u, _ in self.samples)
secs = sum(t for _, t in self.samples)
if not units or not secs:
return float("inf")
return (self.total - self.done) / (units / secs)A moving window tracks the current rate rather than the lifetime average, so an ETA recovers quickly after a slow patch. Weighting by bytes handles the "one 4 GB file among a thousand small ones" case, which no count-based estimate can.
Make the summary the important part
import time
summary = {"total": len(files), "ok": len(ok), "failed": len(failed),
"elapsed_s": round(time.perf_counter() - started, 1)}
summary["rate_per_min"] = round(summary["total"] / (summary["elapsed_s"] / 60), 1)
print("\n─── summary ───")
for k, v in summary.items():
print(f"{k:14} {v}")The bar is for the person watching; the summary is what ends up in the log, the ticket and next week's capacity estimate.
Code examples
Example 1: a complete batch with size-weighted progress
from pathlib import Path
import logging, sys, time
import geopandas as gpd
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(message)s", stream=sys.stdout)
log = logging.getLogger("batch")
SRC, OUT = Path("data/raw"), Path("data/out")
def process_one(path: Path) -> int:
gdf = gpd.read_file(path)
gdf.to_file(OUT / f"{path.stem}.gpkg", driver="GPKG")
return len(gdf)
def main() -> int:
files = sorted(SRC.rglob("*.shp"))
if not files:
log.error("nothing to process in %s", SRC.resolve())
return 2
OUT.mkdir(parents=True, exist_ok=True)
total_bytes = sum(f.stat().st_size for f in files)
log.info("%d files, %.2f GB", len(files), total_bytes / 1e9)
tty = sys.stdout.isatty()
bar = None
if tty:
from tqdm import tqdm
bar = tqdm(total=total_bytes, unit="B", unit_scale=True, desc="converting")
started, done_bytes, ok, failed = time.perf_counter(), 0, 0, []
for i, path in enumerate(files, start=1):
size = path.stat().st_size
try:
process_one(path)
ok += 1
except Exception as exc:
failed.append((path.name, f"{type(exc).__name__}: {exc}"))
(bar.write if bar else log.warning)(f"failed {path.name}: {exc}")
done_bytes += size
if bar:
bar.set_postfix_str(path.name[:28], refresh=False)
bar.update(size)
elif i % 25 == 0 or i == len(files):
elapsed = time.perf_counter() - started
frac = done_bytes / total_bytes
eta = elapsed / frac - elapsed if frac else float("inf")
log.info("[%d/%d] %5.1f%% eta %.1f min", i, len(files), frac * 100, eta / 60)
if bar:
bar.close()
elapsed = time.perf_counter() - started
log.info("%d ok, %d failed in %.1f min", ok, len(failed), elapsed / 60)
for name, err in failed:
log.warning(" ! %s: %s", name, err)
return 1 if failed else 0
if __name__ == "__main__":
raise SystemExit(main())Example 2: nested bars for a two-level job
from tqdm import tqdm
folders = sorted(p for p in Path("data/raw").iterdir() if p.is_dir())
outer = tqdm(folders, unit="folder", desc="regions", position=0)
for folder in outer:
outer.set_postfix_str(folder.name)
files = sorted(folder.glob("*.shp"))
for path in tqdm(files, unit="file", desc=folder.name[:14], position=1, leave=False):
process_one(path)leave=False on the inner bar keeps the display to two lines instead of one per folder.
Example 3: progress for a single long operation
Some steps are one call that takes twenty minutes. Report from inside it.
import rasterio
from tqdm import tqdm
with rasterio.open("data/raw/dem.tif") as src:
windows = list(src.block_windows(1))
profile = src.profile
with rasterio.open("data/out/dem_m.tif", "w", **profile) as dst:
for _, window in tqdm(windows, unit="block", desc="rescaling"):
dst.write((src.read(1, window=window) * 0.3048).astype(profile["dtype"]),
1, window=window)For GDAL and Processing operations, a callback gives you the same thing:
from osgeo import gdal
from tqdm import tqdm
bar = tqdm(total=100, unit="%", desc="warping")
def gdal_progress(complete, message, _):
bar.n = int(complete * 100)
bar.refresh()
return 1
gdal.Warp("data/out/warped.tif", "data/raw/dem.tif",
dstSRS="EPSG:3857", callback=gdal_progress)
bar.close()Example 4: machine-readable progress for a dashboard
import json, time
from pathlib import Path
STATE = Path("logs/progress.json")
def publish(done, total, started, current=""):
elapsed = time.perf_counter() - started
rate = done / elapsed if elapsed else 0
STATE.parent.mkdir(parents=True, exist_ok=True)
STATE.write_text(json.dumps({
"done": done, "total": total, "pct": round(done / total * 100, 1),
"rate_per_s": round(rate, 3),
"eta_s": round((total - done) / rate) if rate else None,
"current": current, "updated": time.time(),
}), encoding="utf-8")A JSON file that another process can poll turns "is it still going?" into a query rather than a guess — and its updated timestamp doubles as a liveness check.
Explanation
Progress reporting looks cosmetic and is really an operational feature. Its job is to answer three questions cheaply: is the job alive, how far has it got, and when will it finish. Each of those changes a decision — whether to wait, whether to kill, whether to start something else — so an hour spent on it pays for itself the first time you avoid killing a run that was nearly done.
The two contexts a batch runs in want opposite things. A terminal can redraw one line, so an animated bar is ideal: it is compact, continuously updated, and disappears when the job ends. A log file cannot — carriage returns produce one monstrous line, and the timestamps that make a log useful are missing. So the switch on sys.stdout.isatty() is not a nicety; it is the difference between a readable journalctl and an unreadable one.
The ETA deserves more thought than it usually gets. Remaining items divided by average rate assumes every item costs the same, which is rarely true for GIS data: file sizes span orders of magnitude, and geometry complexity varies more than that. Weighting progress by bytes is the cheapest good approximation, since bytes correlate with work far better than counts do. A moving window over recent items handles the other distortion — a machine that was slow while a cache warmed, or a run that has just hit a large batch of small files.
Parallel jobs need a different structure again. Workers are separate processes with their own stdout, so they must not draw the bar. The parent, which already collects futures as they complete, is the natural place to count, and tqdm.write() exists precisely so error messages can be printed without corrupting the display.
And whichever style you use, the closing summary is the part that persists. The bar is gone the moment the job ends; the summary line — items, failures, elapsed, rate — is what tells you next month whether the job is getting slower.
Edge cases or notes
flush=Trueor nothing appears: Redirected stdout is block-buffered. Set it per print, or run Python with-u/PYTHONUNBUFFERED=1.- Never
print()inside atqdmloop: Usebar.write(), or the bar and your message will overwrite each other. tqdmwrites to stderr by default: Handy — it keeps stdout clean for real output. Passfile=sys.stdoutif your logging setup expects otherwise.- A generator has no length:
tqdmcannot show a percentage withouttotal=. Materialise the list, or pass the count you already know. - CI logs hate animation: GitHub Actions renders every bar update as a new line. Detect CI (
os.environ.get("CI")) and fall back to periodic lines. tqdmcosts almost nothing: Roughly microseconds per update, but avoid updating it inside a tight inner loop over millions of geometries; update per file instead.- An ETA is an estimate: State it as one. Reporting "eta 12.4 min" to two decimals implies a precision that does not exist.
Internal links
- How to Batch Process a Folder of GIS Files in Python: The Complete Workflow
- How to Log and Summarise Errors in a Batch GIS Job in Python
- How to Build a Resumable Batch GIS Job in Python
- How to Speed Up Batch GIS Jobs with Parallel Processing in Python
- Batch Script Stops at the First Bad File: How to Keep It Running
- My GIS Pipeline Fails Silently: Fixing Swallowed Errors and Wrong Exit Codes
FAQ
Why does my progress bar look broken in a log file?
Because it redraws using carriage returns, which a file records literally. Detect sys.stdout.isatty() and switch to periodic log lines when there is no terminal.
How do I show progress from parallel workers?
Count completions in the parent. Submit futures, iterate as_completed(), and update one bar there — workers should never draw to the shared terminal.
Why is my ETA so inaccurate?
It is probably counting files, not work. Weight progress by file size and compute the rate from a moving window of recent items rather than the whole run.
Do I have to use tqdm?
No. A print every N items with count, percentage and elapsed time covers most needs and has no dependency. tqdm is simply the nicest interactive option.
Why does nothing appear until the job finishes?
Output buffering. Pass flush=True to print, run with python -u, or set PYTHONUNBUFFERED=1 in the environment.
Can I show progress inside a single long GDAL or Processing call?
Yes — both support progress callbacks. gdal.Warp(..., callback=fn) and QgsProcessingFeedback.setProgress() let you drive a bar from inside an operation you did not write.
How do I let something else monitor the job?
Write a small JSON state file every few seconds with done, total, rate and a timestamp. Another process can poll it, and a stale timestamp doubles as a hung-job alert.