forward: sealed forward tests for world models
Every entry is frozen and hashed before a round opens, then scored only on outcomes that resolve after the freeze. Nothing can leak into training and nothing can be tuned after the fact. Each entry reports six columns: a proper score with honest intervals, calibration, skill against copying the last value, skill against the prediction market on the same outcome, paper profit after fees, and how much of any gain one month explains.
The results below are dry runs on past data, through the same pipeline the sealed rounds use. Round 1 freezes on 31 October 2026 and scores November and December; Round 2 freezes on 3 January 2027. Their specs, with every hypothesis and pass rule, are public before any of their data exists.
Task 1: tomorrow's high at 19 weather stations
Each entry gives a probability for every whole degree of the official high at the 19 NWS stations that settle Kalshi's temperature markets, read at 20:00 UTC the day before. The dry run covers 2026-07-01 to 2026-10-09: 1,917 station-days.
| entry | gain | low | high |
|---|---|---|---|
| blend of five, calibrated | +0.758 | +0.702 | +0.813 |
| AIFS (learned), calibrated | +0.623 | +0.567 | +0.677 |
| ICON, calibrated | +0.524 | +0.458 | +0.588 |
| NBM, calibrated | +0.519 | +0.459 | +0.577 |
| IFS (physics), calibrated | +0.489 | +0.435 | +0.540 |
| ICON, raw | +0.427 | +0.374 | +0.478 |
| GFS, calibrated | +0.410 | +0.344 | +0.472 |
| NBM, raw | +0.398 | +0.351 | +0.443 |
| IFS (physics), raw | +0.271 | +0.222 | +0.319 |
| GFS, raw | +0.212 | +0.149 | +0.276 |
| AIFS (learned), raw | +0.210 | +0.157 | +0.264 |
| climatology | -0.208 | -0.251 | -0.167 |
ECMWF's learned model, AIFS, is worse than its physics model, IFS, as it comes out of the box (-0.061, interval -0.092 to -0.029). After the same simple calibration it is the better of the two by +0.133 (+0.089 to +0.178), and the best single model. A blend of all five is best overall, and still loses to the Kalshi market on the same ladders by -0.113 (-0.151 to -0.075), even with the market priced at its bids. Trading it would have made -$0.021 per ladder after fees and a cent of slippage, an interval that includes zero.
| entry | log score | 80% cover | vs market | P&L |
|---|---|---|---|---|
| blend of five, calibrated | -2.155 | 0.79 | -0.113 | -$40 |
| AIFS (learned), calibrated | -2.290 | 0.80 | -0.256 | -$109 |
| ICON, calibrated | -2.388 | 0.82 | -0.339 | -$233 |
| NBM, calibrated | -2.393 | 0.76 | -0.335 | -$327 |
| IFS (physics), calibrated | -2.423 | 0.81 | -0.379 | -$242 |
| ICON, raw | -2.485 | 0.85 | -0.429 | -$283 |
| GFS, calibrated | -2.502 | 0.79 | -0.445 | -$224 |
| NBM, raw | -2.515 | 0.80 | -0.468 | -$369 |
| IFS (physics), raw | -2.641 | 0.86 | -0.572 | -$328 |
| GFS, raw | -2.700 | 0.81 | -0.627 | -$272 |
| AIFS (learned), raw | -2.702 | 0.81 | -0.623 | -$290 |
| copy yesterday | -2.912 | 0.87 | -0.796 | -$307 |
| climatology | -3.121 | 0.82 | -0.987 | -$331 |
log score per station-day, higher is better. 80% cover: share of outcomes inside the central 80% interval (0.80 is calibrated). vs market: log score minus Kalshi's ladder priced at its bids, on the same ladders; every interval excludes zero. P&L: paper taker profit after fees and 1¢ of slippage, one contract per mispriced bucket. Full table with intervals: LEADERBOARD.md.
Task 2: where a price closes
Kalshi lists daily ladders on the S&P 500, the Nasdaq-100, EUR/USD, USD/JPY, WTI, Brent, gold, silver, copper and natural gas. Each entry sees the last 512 hourly prices of the underlying and gives a probability for every bucket, 24 or 6 hours before the close (903 and 1,706 past ladders). The trivial answer is to copy the last price: centre on it, with the spread of past moves over the same hours. The entries are three released time-series foundation models from two families, Chronos-Bolt and TimesFM, used as published.
| entry | gain | low | high |
|---|---|---|---|
| Chronos-Bolt small | -0.046 | -0.100 | +0.010 |
| Chronos-Bolt small, median | -0.041 | -0.073 | -0.012 |
| Chronos-Bolt base | -0.072 | -0.120 | -0.020 |
| Chronos-Bolt base, median | -0.071 | -0.110 | -0.035 |
| TimesFM 2.5 | -0.059 | -0.103 | -0.011 |
| TimesFM 2.5, median | -0.050 | -0.072 | -0.028 |
| Chronos-Bolt small | -0.028 | -0.069 | +0.010 |
| Chronos-Bolt small, median | -0.042 | -0.061 | -0.023 |
| Chronos-Bolt base | -0.020 | -0.056 | +0.018 |
| Chronos-Bolt base, median | -0.053 | -0.074 | -0.032 |
| TimesFM 2.5 | -0.017 | -0.062 | +0.026 |
| TimesFM 2.5, median | -0.035 | -0.053 | -0.020 |
None of them beats copying. Their predicted moves point the right way about half the time (Chronos-Bolt small 49.0%, Chronos-Bolt base 50.1%, TimesFM 2.5 49.1%), and keeping their direction while using the copy baseline's spread makes each one significantly worse. What they add to copying is noise. The copy baseline itself beats the market at its bid by +1.30 a day out, and still does not make money after fees.
How the seal works
- Freeze. Fit every entry on data settled by the cutoff; record the sha256 of the parameters, the spec and every scoring file; push before the first read.
- Capture. At each read, write the live model runs and commit them to a public branch. A capture after 22:00 UTC the day before does not count.
- Score. After the window, refuse if any hash changed; score on truth published after the outcomes; publish the result, including failures.
Code, specs, methods and every result file: github.com/physicalreasoning/forward. Methods: METHODS.md. Full tables: LEADERBOARD.md.