Track record
How the forecasts we have recorded turned out. This includes the ones we got wrong, the ones we could not check and the disasters we missed.
Loading the track record…
Made in advance by our weather models
The only forecasts on this site made before anything was known: unusually hot days, unusually heavy rain and a high fire-weather index, up to 7 days ahead, at 108 cities. Each is recorded when it is published and graded against NASA POWER once its days have been measured and the measurements have settled. These are weather lines, not disasters. How the models work.
Results by hazard
The same figures as the chart above, as a table.
| Hazard | Published | Came true | False alarms | Waiting | Would come true by chance | Missed days |
|---|---|---|---|---|---|---|
| Loading… | ||||||
Day by day, as the backtest is scored
Every day a published run was watching (1 to 7 days ahead, at a place it checked) and that has a settled measurement. This removes the double counting of re-issued forecasts and gives the same rates as the backtest below, with their denominators.
| Hazard | Place-days watched | Line crossed (base rate) | Days forecast | Forecast and crossed | Forecast, not crossed | Crossed, not forecast | Share of crossings forecast | Share of forecast days wrong |
|---|---|---|---|---|---|---|---|---|
| Loading… | ||||||||
Every published model forecast
Each links to the forecast exactly as it was published. Newest first, 25 at a time.
| Forecast | Place | Days covered | Published | Result | What was measured | By chance |
|---|---|---|---|---|---|---|
| Loading… | ||||||
Restated agency alerts: not a test of our skill
Most forecasts on this site restate an alert an agency had already issued (a GDACS alert, a National Weather Service warning, a cyclone advisory) and keep it open for as long as that hazard usually lasts. They are counted once each and graded only on new damage or impact reported inside their period: the agency still listing the hazard does not count.
Experimental forecast models: tested on past weather
Our experimental models forecast unusually hot days, unusually heavy rain and a high fire-weather index up to 7 days ahead. Their forecasts are on the forecasts page, marked as made in advance. Each one is recorded when it is published and graded below once its days have been measured, misses included. The models are also tested on weather forecasts that were really issued 3 and 5 days in advance; those results are in the first table.
Loading the test results…
| Hazard | Forecast made | Days tested | Days it happened | Caught by the model | Caught by chance | Right when the model warned | Right by chance |
|---|
Published model forecasts
Loading the grades…
| Hazard | Published | Came true | False alarms | Missed days | Waiting to be measured |
|---|
Data and reproducibility
Every figure on this page comes from files published with the site. Each model forecast is committed to the ledger before its window starts; the history of that file is the proof it was made in advance.
- Model forecasts as published (JSON) · their grades (JSON) · the NASA POWER measurements they were graded on (JSON) · the backtest (JSON)
- Other forecasts as recorded (JSON, 5 MB) · their verdicts (JSON) · disasters on record (JSON) · this page’s figures (JSON)
- When each model forecast was committed (git history) · the grading code:
forecast-models.js,tools/run-models.js,accuracy-model.js,verification-model.js.node tools/rebuild-model-grades.jsandnode tools/build-accuracy-summary.jsrebuild the grades and these figures from the files.
The full record: every recorded forecast, the disasters, and how each was checked
Opening this downloads about 8 MB of ledgers and grades them in your browser, with the same code that produced the figures above.
What these figures mean
✓ Forecasts that came true
Counted once each. None of these was made in advance: they restate an agency alert or follow an earthquake. Our weather models’ forecasts that came true are in the table above.
⚠ Significant events we missed
Significant disasters on record
Every significant disaster in our record, including the ones we did not forecast. If we have no record of checking that place at the time, we cannot say why a disaster was missed. The table says so, rather than blaming our checks or our forecasts.
| Disaster | Type | Where | Started | Deaths | Displaced | Evidence | Forecast in advance? | Why not |
|---|
All recorded forecasts
One row per forecast, newest first, counted by its first recorded version. You can search, filter or sort. We show 25 at a time.
| Forecast | Did it happen? | Result | Type of forecast | Actual severity | Hazard and place | Locked on | Forecast period | What counted as a match | Matching event | Evidence |
|---|
What we forecast compared with what happened
Each checked forecast, with what we forecast next to what we found.
Full details: advance forecasts, other forecasts and missed disasters
A forecast counts as made in advance only if we recorded it before the danger it names was known. Other forecasts are made later: about a flood that is already happening, about aftershocks after an earthquake, or as a test case written after its period began. These are real forecasts, but they do not foresee something nobody knew about. We report the two kinds separately and never add them together.
How our advance forecasts did
How our other forecasts did
These forecasts are about a danger already reported, or about what follows an event that already happened, such as aftershocks. We check them too, but never count them as advance forecasts.
Disasters and what we missed
How we work out these numbers
How we label results
For each forecast or event, we keep 4 separate labels: how severe the event was, how the forecast turned out, whether we were checking that place, and how strong the evidence is.
Precision answers this question: when we forecast an event, how often did it happen? Recall answers this one: of the significant disasters that really happened, how many had a forecast that counts, made in advance? Forecasts that are still running are left out until their period ends. A forecast based on an official warning gets no credit just because the warning already existed.