How to Analyse an Air Quality Time Series in Python

On this page

Problem statement

A year of hourly NO2 at one station is 8,760 numbers, and the annual mean compresses them into one. Most of what matters is in the structure the mean throws away: the morning rush hour, the weekday–weekend difference, the winter peak, the handful of hours above a legal threshold, and the multi-year trend that says whether any of it is improving.

Each of those has a standard way to compute it, and each has a trap — time zones that move the rush hour, exceedance rules that count hours rather than days, and trends computed over years with different data capture.

Quick answer

Put the series on a proper UTC index, then group by the calendar features that carry the signal:

import io, requests, numpy as np, pandas as pd

def aurn_no2(site, year):
    text = requests.get(f"https://uk-air.defra.gov.uk/datastore/data_files/site_data/"
                        f"{site}_{year}.csv", timeout=120).text
    d = pd.read_csv(io.StringIO(text), skiprows=4, low_memory=False)
    end = pd.to_datetime(d.Date, dayfirst=True) + pd.to_timedelta(d.time.str[:2].astype(int), "h")
    start = (end - pd.Timedelta("1h")).dt.tz_localize("UTC")
    return pd.Series(pd.to_numeric(d["Nitrogen dioxide"], errors="coerce").to_numpy(),
                     index=start, name=site)

no2 = aurn_no2("MY1", 2023)
local = no2.tz_convert("Europe/London")
weekday = local.index.dayofweek < 5
profile = pd.DataFrame({
    "weekday": local[weekday].groupby(local[weekday].index.hour).mean(),
    "weekend": local[~weekday].groupby(local[~weekday].index.hour).mean()})
print("Marylebone Road NO2 by local hour (µg/m³)")
print(profile.loc[[3, 6, 8, 12, 17, 20, 23]].round(1).to_string())
print(f"\nweekday mean {local[weekday].mean():.1f}, weekend mean {local[~weekday].mean():.1f} µg/m³")

Marylebone Road NO2 by local hour (µg/m³)
    weekday  weekend
3      21.5     24.3
6      35.2     26.9
8      50.5     33.7
12     51.7     41.2
17     57.9     49.4
20     46.4     45.6
23     39.1     39.1

weekday mean 43.3, weekend mean 37.5 µg/m³

The diurnal profile is a fingerprint of the source. A traffic site shows its morning and evening peaks on weekdays and a flatter, later profile at weekends.

UTC storage, weekdays, seasons, regulatory exceedances

1. Index by UTC, analyse in local time

Store the series in UTC so that joins and resampling are unambiguous. Convert to local civil time only for anything tied to human activity — rush hours move by an hour twice a year in UTC.

2. Separate weekdays from weekends

Traffic emissions follow the working week. A profile averaged over all days blurs two different patterns into one.

3. Look at the seasonal cycle

NO2 tends to be higher in the cold half of the year: the boundary layer is shallower, there is less sunlight to break NO2 down, and heating adds emissions. Ozone does the opposite. Individual years vary — at Marylebone Road in 2023 the worst month was October and the cleanest May.

4. Count exceedances the way the rule defines them

The EU NO2 hourly limit is 200 µg/m³ not to be exceeded more than 18 times a year — a count of hours. The PM10 daily limit is 50 µg/m³ on no more than 35 days — a count of valid daily means. Using the wrong unit of counting changes the answer.

5. Compare years only with similar data capture

A year with 60% capture has a mean biased by which months are missing. Check capture per year before drawing a trend line.

6. Fit trends on annual means, not hours

Hourly data are autocorrelated and seasonal; a regression on 50,000 hours gives a wildly overconfident slope. Use annual (or deseasonalised monthly) means.

Weekday rush hours and a later, lower weekend: the traffic signature.

The week, the year and a multi-year trend

The data used on this page

Hourly measurements from the UK's Automatic Urban and Rural Network, downloaded by the page's own code from DEFRA's site-data archive: London Marylebone Road (site MY1), a kerbside station. The files are ratified data; the hour in each row is the end of the measurement hour, in GMT.

import requests

reply = requests.head("https://uk-air.defra.gov.uk/datastore/data_files/site_data/MY1_2023.csv", timeout=60,
                      allow_redirects=True)
print("MY1_2023.csv:", reply.status_code)

MY1_2023.csv: 200

Example 1 — the week and the year

by_day = local.groupby(local.index.dayofweek).mean()
by_day.index = ["Mon", "Tue", "Wed", "Thu", "Fri", "Sat", "Sun"]
print("by weekday:", by_day.round(1).to_dict())
by_month = local.groupby(local.index.month).mean()
print(f"highest month {by_month.idxmax()} ({by_month.max():.1f}), "
      f"lowest month {by_month.idxmin()} ({by_month.min():.1f}) µg/m³")

by weekday: {'Mon': 41.0, 'Tue': 43.0, 'Wed': 44.7, 'Thu': 43.8, 'Fri': 44.0, 'Sat': 39.2, 'Sun': 35.9}
highest month 10 (50.9), lowest month 5 (32.1) µg/m³

Sunday is the cleanest day of the week at a traffic site, 20% below Wednesday — the clearest single piece of evidence that the road is the source.

Example 2 — exceedances under the actual rules

def pm10_series(site, year):
    text = requests.get(f"https://uk-air.defra.gov.uk/datastore/data_files/site_data/"
                        f"{site}_{year}.csv", timeout=120).text
    d = pd.read_csv(io.StringIO(text), skiprows=4, low_memory=False)
    v = pd.to_numeric(d["PM<sub>10</sub> particulate matter (Hourly measured)"], errors="coerce")
    return v.groupby(d.Date).agg(["mean", "count"])

pm10 = pm10_series("MY1", 2023)
valid_days = pm10[pm10["count"] >= 18]
print(f"NO2 hours above 200 µg/m³: {(no2 > 200).sum()} (limit: 18)")
print(f"NO2 hours above 100 µg/m³: {(no2 > 100).sum()}")
print(f"PM10 valid days: {len(valid_days)}; days above 50 µg/m³: "
      f"{(valid_days['mean'] > 50).sum()} (limit: 35)")
print(f"PM10 days above 45 µg/m³ (WHO 24-h guideline): {(valid_days['mean'] > 45).sum()}")

NO2 hours above 200 µg/m³: 0 (limit: 18)
NO2 hours above 100 µg/m³: 68
PM10 valid days: 340; days above 50 µg/m³: 1 (limit: 35)
PM10 days above 45 µg/m³ (WHO 24-h guideline): 3

A site can comply with every short-term limit and still exceed the annual one — which is Marylebone Road in 2023. Short-term limits guard against episodes; the annual mean measures chronic exposure.

Example 3 — a multi-year trend

years = range(2015, 2024)
annual = []
for y in years:
    s = aurn_no2("MY1", y)
    annual.append({"year": y, "mean": s.mean(), "capture": s.notna().mean()})
annual = pd.DataFrame(annual).set_index("year")
print(annual.round({"mean": 1, "capture": 2}).to_string())
ok = annual[annual.capture >= 0.75]
slope, intercept = np.polyfit(ok.index, ok["mean"], 1)
print(f"\ntrend over {len(ok)} years with ≥75% capture: {slope:+.1f} µg/m³ per year")

      mean  capture
year               
2015  88.3     0.99
2016  89.1     0.99
2017  83.7     0.99
2018  84.7     0.98
2019  62.7     0.95
2020  43.7     0.97
2021  43.0     0.94
2022  44.6     0.99
2023  41.6     0.95

trend over 9 years with ≥75% capture: -7.4 µg/m³ per year

Marylebone Road's NO2 more than halved, from 88 to 42 µg/m³ — but not along a line. It was flat around 84–89 until 2018, dropped in 2019, the year central London's Ultra Low Emission Zone began in April, and has been flat around 42–45 since 2020. The fitted −7.4 µg/m³ per year describes none of those years; the honest summary is a step. Plot the annual means before fitting anything to them.

A step, not a slope: flat, a drop in 2019, flat again.

Diurnal profiles, exceedance rules and autocorrelation

Why the diurnal profile is diagnostic

Emissions, dilution and chemistry all have daily cycles. Traffic peaks at rush hours; the boundary layer is shallow at night and in the early morning, trapping emissions; sunlight photolyses NO2 in the afternoon. The combination produces a profile characteristic of each source type, which is why air quality analysts look at profiles before they look at maps.

Why exceedance rules count different things

Short-term limits were set to protect against acute effects, which depend on peak exposure, so they count hours or days above a threshold with an allowance for rare events. Long-term limits protect against chronic effects, so they use the mean. Each rule is a statistic of its own, and "the site exceeded the limit" is incomplete until it says which.

Why trends need care with autocorrelation

Consecutive hours are nearly the same number, and the seasonal cycle repeats every year. A regression on hourly values treats every hour as independent evidence and reports a slope with a vanishingly small error. Annual means remove both problems at the cost of having few points, which is an honest cost.

Short-term limits guard against episodes; the annual mean measures chronic exposure.

Daylight saving, holidays and 2020

Daylight saving: in UTC the rush hour shifts by an hour at the change; in local time the data gains and loses an hour.
Bank holidays behave like weekends; include them in the weekend group for sharper profiles.
COVID-19 in 2020 makes that year an outlier for traffic pollutants; say whether it is in the trend.
Instrument changes (e.g. BAM to FIDAS for particles) can create steps in a long series.
Deseasonalised trends: for shorter periods, fit on monthly anomalies from the multi-year monthly mean.

Internal links

How to download air quality station data in Python — the loader behind this guide
Air pollutants and their units explained — limits and their averaging periods
Air quality data gaps and negative values — capture before trends
How to make a pollution rose — adding wind direction to the time series
How to resample a time series with xarray — the same operations on gridded data
Solar timestamps shifted by an hour — time conventions in another domain

FAQ

How do I plot a diurnal profile of air pollution?

Convert the index to local time, group by hour of day, and average — separately for weekdays and weekends.

Why is NO2 higher in winter?

A shallower mixing layer, less sunlight to break it down, and more emissions from heating.

How are NO2 hourly exceedances counted?

Hours above 200 µg/m³ in a calendar year; the EU allows 18.

How are PM10 daily exceedances counted?

Valid daily means above 50 µg/m³; the EU allows 35 days a year.

How do I compute a pollution trend?

Fit on annual means from years with adequate data capture, or on deseasonalised monthly means.

Why is Sunday the cleanest day at traffic sites?

Traffic volumes, especially of goods vehicles, are lowest on Sundays.