Track record

How the forecasts we have recorded turned out. This includes the ones we got wrong, the ones we could not check and the disasters we missed.

Loading the track record…

Made in advance by our weather models

The only forecasts on this site made before anything was known: unusually hot days, unusually heavy rain and a high fire-weather index, up to 7 days ahead, at 108 cities. Each is recorded when it is published and graded against NASA POWER once its days have been measured and the measurements have settled. These are weather lines, not disasters. How the models work.

Published in advance-
Graded so far-
Came true-
False alarms-
Would come true by chance-
Missed days-
Waiting for settled measurements-
Claims, re-issues merged-

Results by hazard

The same figures as the chart above, as a table.

HazardPublishedCame trueFalse alarmsWaitingWould come true by chanceMissed days
Loading…

Day by day, as the backtest is scored

Every day a published run was watching (1 to 7 days ahead, at a place it checked) and that has a settled measurement. This removes the double counting of re-issued forecasts and gives the same rates as the backtest below, with their denominators.

HazardPlace-days watchedLine crossed (base rate)Days forecastForecast and crossedForecast, not crossedCrossed, not forecastShare of crossings forecastShare of forecast days wrong
Loading…

Every published model forecast

Each links to the forecast exactly as it was published. Newest first, 25 at a time.

ForecastPlaceDays coveredPublishedResultWhat was measuredBy chance
Loading…

Restated agency alerts: not a test of our skill

Most forecasts on this site restate an alert an agency had already issued (a GDACS alert, a National Weather Service warning, a cyclone advisory) and keep it open for as long as that hazard usually lasts. They are counted once each and graded only on new damage or impact reported inside their period: the agency still listing the hazard does not count.

Forecasts-
Could be checked-
Correct-
Partly correct-
Wrong-
Cannot be checked-

Experimental forecast models: tested on past weather

Our experimental models forecast unusually hot days, unusually heavy rain and a high fire-weather index up to 7 days ahead. Their forecasts are on the forecasts page, marked as made in advance. Each one is recorded when it is published and graded below once its days have been measured, misses included. The models are also tested on weather forecasts that were really issued 3 and 5 days in advance; those results are in the first table.

Loading the test results…

Published model forecasts

Loading the grades…

Data and reproducibility

Every figure on this page comes from files published with the site. Each model forecast is committed to the ledger before its window starts; the history of that file is the proof it was made in advance.

The full record: every recorded forecast, the disasters, and how each was checked

Opening this downloads about 8 MB of ledgers and grades them in your browser, with the same code that produced the figures above.

Forecasts (one per forecast)-
Still running-
Ended, not checked yet-
Ended, cannot be checked-
Checked so far-
Correct-
Partly correct-
Wrong (false alarm)-
Missed events-
No proof made in time-
What these figures mean

✓ Forecasts that came true

Counted once each. None of these was made in advance: they restate an agency alert or follow an earthquake. Our weather models’ forecasts that came true are in the table above.

Shown when the full record is open.

⚠ Significant events we missed

Shown when the full record is open.

Significant disasters on record

Every significant disaster in our record, including the ones we did not forecast. If we have no record of checking that place at the time, we cannot say why a disaster was missed. The table says so, rather than blaming our checks or our forecasts.

DisasterTypeWhereStartedDeathsDisplacedEvidenceForecast in advance?Why not

All recorded forecasts

One row per forecast, newest first, counted by its first recorded version. You can search, filter or sort. We show 25 at a time.

ForecastDid it happen?ResultType of forecastActual severityHazard and placeLocked onForecast periodWhat counted as a matchMatching eventEvidence
What we forecast compared with what happened

Each checked forecast, with what we forecast next to what we found.

Shown when the full record is open.
Full details: advance forecasts, other forecasts and missed disasters

A forecast counts as made in advance only if we recorded it before the danger it names was known. Other forecasts are made later: about a flood that is already happening, about aftershocks after an earthquake, or as a test case written after its period began. These are real forecasts, but they do not foresee something nobody knew about. We report the two kinds separately and never add them together.

Forecasts recorded-
Forecast periods covered-
First advance forecast-
Last updated-

How our advance forecasts did

Advance forecasts-
Checked-
Correct-
Partly correct-
Wrong (false alarm)-
Still running or cannot be checked-
No proof made in time-
Precision: how often right-
Recall: share of disasters forecast-
F1 score: the two combined-

How our other forecasts did

These forecasts are about a danger already reported, or about what follows an event that already happened, such as aftershocks. We check them too, but never count them as advance forecasts.

Other forecasts-
Checked-
Correct-
Partly correct-
Wrong (false alarm)-
Still running or cannot be checked-
Precision: how often right-

Disasters and what we missed

Significant disasters-
Not forecast in advance-
Missed: not checking there-
Missed by our forecasts-
Reason unknown-
Cannot be forecast-
No advance source-

How we work out these numbers

How we label results

For each forecast or event, we keep 4 separate labels: how severe the event was, how the forecast turned out, whether we were checking that place, and how strong the evidence is.

Extreme eventSevere eventSignificant eventCorrectPartly correctWrong (false alarm)Missed eventMissed: not checking thereCannot be checkedNo proof it was made in time

Precision answers this question: when we forecast an event, how often did it happen? Recall answers this one: of the significant disasters that really happened, how many had a forecast that counts, made in advance? Forecasts that are still running are left out until their period ends. A forecast based on an official warning gets no credit just because the warning already existed.