A benchmark you can't train on

Every benchmark leaks eventually. Questions end up in training data, people tune against the test set, and a high score stops meaning what it did. Outcomes that have not happened yet cannot leak. So we built a benchmark that only scores models on the future: every entry is frozen and hashed before a round opens, and graded on what actually happens afterwards. The first sealed round freezes on 31 October.

It is called forward, and the dry runs on past data already say two clear things. ECMWF's learned weather model loses to its physics model as it comes out of the box, and beats it once both get the same simple calibration. And two families of released time-series foundation models cannot beat the most boring forecast there is: copy the last price.

Six columns, no exceptions

A single headline number hides most of what matters, so every entry reports the same six things: a proper score (log score and ranked probability score) with honest intervals; calibration; skill against copying the last value and against climatology; skill against the prediction market that prices the same outcome; paper profit after fees and slippage; and how much of any gain one calendar month explains.

Two of these come straight from our own mistakes. We once trained self-supervised models on market data that learned to copy their input, while every standard diagnostic said they were healthy (write-up). So beating the copy is mandatory. And in our weather market work, a better score repeatedly failed to turn into money once the spread and fees were paid. So the market, priced at its bids, and the profit column are mandatory too.

The learned model needs calibrating

The first task is the official high temperature tomorrow at the 19 weather stations that settle Kalshi's temperature markets, forecast at 20:00 UTC the day before. Each entry is a probability for every whole degree. The inputs are five weather models served free by Open-Meteo: ECMWF's learned model AIFS and its physics model IFS, NOAA's GFS, DWD's ICON and NOAA's blended NBM. Each is entered twice, once raw and once with a standard correction, EMOS: a linear fix for bias and a spread fitted per station on past data. The dry run covers 1 July to 9 October 2026: 1,917 station-days.

Every model beats copying yesterday, which is reassuring and not interesting. What is interesting is the order. Raw, AIFS is slightly worse than IFS (−0.061, interval −0.092 to −0.029). Calibrated, it is clearly better (+0.133, interval +0.089 to +0.178), and the best single model. The reason is simple: over these days AIFS runs about 2.1°F cold on average, while IFS is nearly unbiased. Take out each station's bias and AIFS's remaining error is 2.3°F against IFS's 2.8°F (root mean square). The ranking of the two models flips on a step most comparisons leave out.

The market is another matter. A blend of all five calibrated models is the best entry, and on the same ladders it still loses to the Kalshi market priced at its bids (−0.113, interval −0.151 to −0.075). Trading on it would have made −$0.021 (interval −$0.061 to +$0.019) per ladder after fees and a cent of slippage, an interval that includes zero. Free forecasts are already in the price.

Foundation models do not beat copying

The second task is where a price closes. Kalshi lists daily ladders on the S&P 500, the Nasdaq-100, two currency pairs, oil, gold, silver, copper and natural gas. Each entry sees the last 512 hourly prices of the underlying and gives a probability for every bucket, 24 or 6 hours before the close (903 and 1,706 past ladders). The baseline copies the last price: centre on it, with the spread of past moves over the same hours. Against it we entered Chronos-Bolt (Amazon) and TimesFM 2.5 (Google), used exactly as released.

None of them is above zero. Their predicted moves point the right way about half the time (Chronos-Bolt small 49%, Chronos-Bolt base 50%, TimesFM 2.5 49%), and when we keep only their direction, every one of them gets significantly worse. Whatever these models learned about time series, on prices close to a random walk what they add to copying is noise. Our own models failed the other way: they learned to copy and still looked healthy. These models move away from the copy and score worse for it. Either way, the copy column is what catches it.

What would an action have done?

A world model should also answer counterfactuals: what would have happened if we had acted. That is hard to test, because you usually have to perform the action. Markets are an exception. We record Kalshi's order books ourselves, and a matching engine is deterministic, so replaying the record shows what a limit order we never placed would have done: whether it filled, and where the price went next. The third task scores models on exactly that. One day of recordings is a pipeline check, not a result, but it already turned up a design decision: more than half of the hypothetical fills come from orders crossing the book rather than from trades, so a sealed version has to say in advance which counts.

What is sealed

Round 1 freezes on 31 October and scores 1 November to 26 December. Its five tests are written down: whether calibrated AIFS beats calibrated IFS, whether every calibrated model beats copying, whether the best blend beats the market at its bids, whether trading it makes money, and whether isotherm, our own model of these markets, beats the market. isotherm is entered as it runs, its answers captured live from its public API at every read. Round 2 freezes on 3 January and adds the price task. Every result gets published, whichever way it goes.

Code, specs, methods and every result file: physicalreasoning/forward. Results page: physicalreasoning.ai/forward. Weather data by Open-Meteo.com (CC BY 4.0), from ECMWF (© ECMWF, CC BY 4.0), DWD (CC BY 4.0) and NOAA. Truth from NWS climate reports via the Iowa Environmental Mesonet. Market data is used for scoring and not republished.

References. T. Gneiting, A. E. Raftery, A. H. Westveld, T. Goldman, Calibrated probabilistic forecasting using ensemble model output statistics and minimum CRPS estimation, Monthly Weather Review, 2005. S. Lang et al., AIFS: ECMWF's data-driven forecasting system, 2024. A. F. Ansari et al., Chronos: learning the language of time series, TMLR, 2024. A. Das et al., A decoder-only foundation model for time-series forecasting, ICML, 2024. R. Cont, S. Stoikov, R. Talreja, A stochastic model for order book dynamics, Operations Research, 2010.