🔧 Fix Wildfire mapping Data cleaning Notebook By · Updated

Why Are My Daily Fire Detection Counts on the Wrong Day?

On this page

Problem statement

Satellite fire detections are timestamped in UTC, and files are published by date. Neither date is the day a person on the ground lived through. GOES-East first saw the 2024 Park Fire at 15:31 on 24 July, California time. Count detections by UTC date and almost nothing happened on the 24th, with the fire's first evening and night booked to the 25th. Count them by the file they came in, and you get a third answer, because a NOAA HMS fire file is not a UTC day either. A daily chart built either way shifts the fire's activity by up to a day. A gap in one file can make a quiet day out of a busy one.

This page counts the Park Fire's GOES-East detections in NOAA's Hazard Mapping System (HMS) by file, by UTC date and by Pacific date. It shows where each boundary falls and how to put days where they belong.

Quick answer

Build a UTC timestamp from the detection's own fields, convert it to the fire's time zone, and group by the local date, never by the file:

goes = park[park.source == "GOES-EAST FDC"]
local = goes.time.dt.tz_convert("America/Los_Angeles")
days = pd.DataFrame({"by file": goes.groupby("file").size(),
                     "by UTC date": goes.groupby(goes.time.dt.strftime("%d %b")).size(),
                     "by Pacific date": goes.groupby(local.dt.strftime("%d %b")).size()}).fillna(0).astype(int)
print(days.loc["24 Jul":"27 Jul"].T.to_string())
                 24 Jul  25 Jul  26 Jul  27 Jul
by file            1178    8119    1331    1134
by UTC date          28    6632    3056    2013
by Pacific date    1545    7752    1757     727
Bar chart of the Park Fire's GOES-East detections per Pacific hour from noon on 24 July to midnight on 27 July: a first peak around midnight on 24 to 25 July, a larger peak in the afternoon and evening of 25 July, a seven-hour gap with no data from 23:00 on 25 July, and a smaller peak on the evening of 26 July; vertical lines mark Pacific midnights, UTC midnights at 17:00 and HMS file breaks at 23:00, each cutting through the busiest hours. Bar chart of the Park Fire's GOES-East detections per Pacific hour from noon on 24 July to midnight on 27 July: a first peak around midnight on 24 to 25 July, a larger peak in the afternoon and evening of 25 July, a seven-hour gap with no data from 23:00 on 25 July, and a smaller peak on the evening of 26 July; vertical lines mark Pacific midnights, UTC midnights at 17:00 and HMS file breaks at 23:00, each cutting through the busiest hours.
Each kind of day boundary falls somewhere different in the fire's daily cycle, and UTC midnight, at 17:00 Pacific, lands in the middle of the afternoon peak.

What a file holds, where day one went, UTC handling

1. Check what a file holds

Look at the earliest and latest detection time in each file, by sensor:

win = hms[hms.file.isin(["24 Jul", "25 Jul", "26 Jul"])].groupby(["file", "Method"]).time.agg(["min", "max"])
print(win.apply(lambda c: c.dt.strftime("%d %b %H:%M")).to_string())
                          min           max
file   Method                              
24 Jul ANALYSIS  24 Jul 06:28  25 Jul 04:30
       FDC       24 Jul 06:00  25 Jul 05:56
       VIIRS     24 Jul 05:37  24 Jul 23:01
25 Jul ANALYSIS  25 Jul 06:09  26 Jul 08:20
       FDC       25 Jul 06:00  26 Jul 05:56
       VIIRS     25 Jul 06:09  25 Jul 23:27
26 Jul ANALYSIS  26 Jul 08:08  27 Jul 07:31
       FDC       26 Jul 13:11  27 Jul 05:56
       VIIRS     26 Jul 05:50  26 Jul 23:08

The GOES detections in a file run from 06:00 UTC on its date to 05:56 UTC the next day. The VIIRS detections stay within the UTC date. A file is a processing unit, not a calendar day in any time zone.

2. See where the fire's first day went

The first detection, in UTC and in Pacific daylight time:

first = goes.time.min()
print(f"first GOES-East detection: {first:%d %b %H:%M} UTC = {first.tz_convert('America/Los_Angeles'):%d %b %H:%M} Pacific")
evening = goes[(local.dt.date == pd.Timestamp("2024-07-24").date())]
print(f"Pacific 24 Jul detections: {len(evening):,}, of which {(evening.time.dt.day == 25).sum():,} carry a UTC date of 25 Jul")
first GOES-East detection: 24 Jul 22:31 UTC = 24 Jul 15:31 Pacific
Pacific 24 Jul detections: 1,545, of which 1,517 carry a UTC date of 25 Jul

Pacific time is UTC−7 in summer, so everything after 17:00 local carries the next UTC date. A fire that burns hardest in the afternoon and evening has most of each day's activity split across two UTC dates.

3. Do not localise UTC times

A common mistake is to parse the time without a zone and then call tz_localize with the local zone. That labels UTC clock times as local ones, shifting every detection by seven hours:

naive = pd.to_datetime(goes.YearDay.astype(str) + goes.Time.astype(str).str.zfill(4), format="%Y%j%H%M")
wrong = naive.dt.tz_localize("America/Los_Angeles")
print(f"wrong first detection: {wrong.min():%d %b %H:%M %Z}; right: {local.min():%d %b %H:%M %Z}")
wrong first detection: 24 Jul 22:31 PDT; right: 24 Jul 15:31 PDT

Parse with utc=True (or tz_localize("UTC")), then tz_convert to the local zone.

Grouped bar chart of Park Fire GOES-East detections for 24 to 27 July counted three ways: by file 1,178, 8,119, 1,331 and 1,134; by UTC date 28, 6,632, 3,056 and 2,013; by Pacific date 1,545, 7,752, 1,757 and 727. Grouped bar chart of Park Fire GOES-East detections for 24 to 27 July counted three ways: by file 1,178, 8,119, 1,331 and 1,134; by UTC date 28, 6,632, 3,056 and 2,013; by Pacific date 1,545, 7,752, 1,757 and 727.
The same detections give three different daily series. By UTC date the fire barely existed on 24 July.

4. Check for gaps before comparing days

A day with fewer detections may just be a day with less data. Count the hours with any GOES-East detection anywhere in North America, by Pacific date:

allg = hms[hms.source == "GOES-EAST FDC"]
hours = allg.time.dt.tz_convert("America/Los_Angeles").dt.floor("h")
cover = hours.groupby(hours.dt.strftime("%d %b")).nunique()
print(cover.loc["24 Jul":"27 Jul"].to_string())
gap = pd.date_range("2024-07-24", "2024-07-28", freq="h", tz="America/Los_Angeles", inclusive="left").difference(hours.unique())
print(f"{len(gap)} hours with no GOES-East detection anywhere: {gap.min():%d %b %H:%M} to {gap.max():%d %b %H:%M} Pacific")
time
24 Jul    24
25 Jul    23
26 Jul    18
27 Jul    24
7 hours with no GOES-East detection anywhere: 25 Jul 23:00 to 26 Jul 05:00 Pacific

The 26 July file has no GOES-East detections before 13:11 UTC, so seven overnight hours are missing. Any day that contains them, whether the Pacific date 26 July or the fire day starting 05:00 on 25 July, is undercounted and cannot be compared with its neighbours.

A fire day that starts at 05:00

The data used on this page

NOAA Hazard Mapping System fire point text files dated 23–28 July 2024, which together cover Pacific dates 24–27 July, and the Park Fire perimeter from CAL FIRE's last-five-years perimeter service, projected to California Albers (EPSG:3310).

from pathlib import Path

import geopandas as gpd
import pandas as pd
import requests

Path("data").mkdir(exist_ok=True)
URL = "https://satepsanone.nesdis.noaa.gov/pub/FIRE/web/HMS/Fire_Points/Text/2024/07/"
frames = []
for d in pd.date_range("2024-07-23", "2024-07-28"):
    f = Path(f"data/hms_fire{d:%Y%m%d}.txt")
    if not f.exists():
        f.write_bytes(requests.get(URL + f.name, timeout=120).content)
    t = pd.read_csv(f, skipinitialspace=True)
    t.columns = t.columns.str.strip()
    frames.append(t.assign(file=f"{d:%d %b}"))
hms = pd.concat(frames, ignore_index=True)
hms["Method"] = hms.Method.str.strip()
hms["source"] = hms.Satellite.str.strip() + " " + hms.Method
hms["time"] = pd.to_datetime(hms.YearDay.astype(str) + hms.Time.astype(str).str.zfill(4), format="%Y%j%H%M", utc=True)
perims = gpd.read_file(requests.get("https://services1.arcgis.com/jUJYIo9tSA7EHvfZ/arcgis/rest/services/California_Fire_Perimeters_last_5_years_View/FeatureServer/0/query",
                                    params={"where": "FIRE_NAME='PARK' AND YEAR_=2024", "outFields": "FIRE_NAME", "outSR": 4326, "f": "geojson"}, timeout=120).text)
PARK = perims.to_crs(3310).geometry.make_valid().union_all()
pts = gpd.GeoDataFrame(hms, geometry=gpd.points_from_xy(hms.Lon, hms.Lat), crs=4326).to_crs(3310)
park = pts[pts.within(PARK.buffer(1000))]
print(f"{len(hms):,} HMS fire points in 6 files; {len(park):,} within 1 km of the Park Fire")
236,426 HMS fire points in 6 files; 26,363 within 1 km of the Park Fire

Example 1: a fire day that starts at 05:00

Fire activity is lowest around dawn, so a day running from 05:00 to 05:00 local keeps each afternoon and the night after it together:

by_hour = local.dt.hour.value_counts().sort_index()
print(f"quietest Pacific hour: {by_hour.idxmin():02d}:00 ({by_hour.min()} detections over the period)")
fire_day = (local - pd.Timedelta(hours=5)).dt.strftime("%d %b")
print(goes.groupby(fire_day).size().rename("detections, fire day 05:00–05:00 Pacific").loc["24 Jul":"27 Jul"].to_string())
quietest Pacific hour: 05:00 (142 detections over the period)
time
24 Jul    2893
25 Jul    6404
26 Jul    2264
27 Jul     296

Three clocks in play

Three clocks are in play. Satellites record UTC. Distributors split their files at whatever time suits their processing: HMS fire files break at 06:00 UTC for GOES. People, and fire agencies, live in local time, where the burning period runs from late morning into the night. A day boundary in the wrong place does not lose detections, but it moves them, and a fire's daily rhythm puts the boundary right in the middle of its most active hours.

Four ways to define a day for Park Fire detections, and when each starts in Pacific daylight time. where each "day" starts, in Pacific timeday defined bystartsPark Fire, 24 JulUTC date17:00 the day before28 detectionsHMS file (GOES rows)23:00 the day before1,178Pacific calendar date00:001,545fire day05:00, the quietest hour2,893GOES-East detections in HMS; first detection 15:31 Pacific on 24 July spatialworkflow.io
Where each kind of day starts, and what it makes of the fire's first day.

The fix is a habit: keep every timestamp in UTC internally, convert once for reporting, and choose the reporting day deliberately. For daily totals of fire activity, a local calendar day is understandable, while a fire day starting at the quiet hour before dawn keeps each burning period whole. Whichever is used, state it, and check that every day has full coverage before comparing one with another.

Daylight saving, zones and duplicates

Daily fire detection count problems and fixes. day boundary problemsSymptomFixthe first evening lands on day twoconvert UTC to local beforegroupingevery time 7 hours latetz_convert UTC times; nevertz_localizecounts differ by filegroup by YearDay + Time, not fileone day suddenly quietcount hours covered beforecomparingNOAA HMS fire points, Park Fire, July 2024 spatialworkflow.io
Day boundary problems and fixes.
  • Daylight saving time: tz_convert with a named zone (America/Los_Angeles) handles the switch; a fixed -07:00 offset does not.
  • Fires crossing zones: a fire spanning two time zones needs one reporting zone chosen for it.
  • Smoke polygons are drawn only from daylight imagery, so their files hold daytime windows within the UTC date.
  • Duplicates: a few detections appear in two consecutive files (36 here); drop duplicates on location, time and satellite.
  • GOES-West vs GOES-East: both are in HMS, with different scan times and pixel footprints; count them separately.

FAQ

What time zone are satellite fire detections in?

UTC. HMS, FIRMS and the GOES fire product all record times in UTC.

Does an HMS fire file cover one UTC day?

No. In the July 2024 files the GOES detections run from 06:00 UTC on the file's date to 05:56 UTC the next day. Group by the detection times, not the file.

How do I convert detection times to local time in pandas?

Parse with pd.to_datetime(..., utc=True) and then .dt.tz_convert("America/Los_Angeles"). Do not use tz_localize on times that are already UTC.

What is a fire day?

A reporting day that starts at the quietest hour of fire activity, usually around dawn, so each afternoon and the night after it fall on the same day.