The weather market learned to read the forecast
Kalshi lets you trade tomorrow's high temperature in seven US cities. The National Weather Service publishes its forecast for free, hours before most of the trading happens. We wanted to know whether the market uses it. In 2023 it mostly did not. By 2025 it had priced in the main forecast, and the edge that remained kept shrinking while we measured it.
Every day, for every city, Kalshi lists a ladder of 2°F buckets and exactly one of them settles YES. Each ladder is a small classification problem with a price on every class, so it is a clean setup for asking whether a model's probabilities beat the market's. We pulled every settled ladder back to 2021 (about 8,200 city-days and 23,000 ladder snapshots), every trade since mid 2023 (17 million prints) and twelve years of archived NWS forecasts, all from free public APIs.
The model is called isotherm, and its design follows Jev, TypeSafe's decision model: you give it a state and a typed question, and it gives back calibrated probabilities instead of text. It puts one probability distribution over the outcome and reads every answer off that, so "which bucket wins" and "will it be above 85" cannot contradict each other. The model is small: a learned log pool of the market and the forecasts plus an MLP correction, in an ensemble of five, trained on log loss with recent days weighted more heavily. It starts out identical to the market and moves away from it only where that lowers the loss.
Most of the work went into the evaluation. Everything is walk-forward: each forecast is the version that was public at the moment of the read, each model is refit every quarter on the past only, and the last three months were sealed until the end. We score with proper scoring rules (log loss, RPS, Brier) against the market's implied probabilities at the same timestamp. As a control, we trained the same network on labels sampled from the market itself, which can only beat the market if the pipeline leaks the answer. It scored within 0.002 nats of the market.
The market caught up
In late 2023, blending the market with the free forecast beat the market by 0.18 nats per ladder, a large gain for a liquid market. By 2025 the same blend was at zero, then slightly negative, so traders had started using the forecast. isotherm stays positive in every half year, but it decays too, down to about 0.01.
| half | simple pool | isotherm |
|---|---|---|
| 2023 H2 | +0.182 | +0.227 |
| 2024 H1 | +0.113 | +0.179 |
| 2024 H2 | +0.089 | +0.098 |
| 2025 H1 | -0.012 | +0.022 |
| 2025 H2 | -0.004 | +0.016 |
| 2026 H1 | -0.021 | +0.006 |
The NWS puts out two main public forecast products. NBM is the blend behind the forecast on weather.gov, the one most people see. GFS MOS is older and plainer and mostly lives in text bulletins. Over the last twelve months, adding NBM to the market price does not help at any read time, and at 8am it makes the price worse. Adding GFS MOS still helps at every read time, with confidence intervals clear of zero. The likely reason is visibility: traders priced in the forecast on the website and not the one in the bulletins.
| read | NBM | GFS MOS |
|---|---|---|
| 4pm day before | -0.004 | +0.020 |
| 8am | -0.021 | +0.012 |
| noon | -0.000 | +0.009 |
From probabilities to profit
Better probabilities do not automatically make money, because spreads and fees take most of the edge. Taking liquidity with the simple blends made about one cent per contract, and one extra cent of slippage turned that negative. Making liquidity earned the spread back, but our resting orders mostly filled when the price was moving against us, which is adverse selection. The model's information cut that loss by two thirds compared with quoting blind, but not to zero.
The version that worked trades at 4pm the day before, only when the GFS-adjusted price disagrees with the market by at least four cents, sized with fractional Kelly over the whole ladder at once. Out of sample, with each quarter's settings chosen only on earlier quarters, it made $22k on a $10k paper bankroll over three years.
| Sharpe, annualised (95% bootstrap CI) | 2.05 [0.81, 3.27] |
| Newey-West t on daily PnL | 3.17 |
| max drawdown | $5,195 |
| hit rate | 71% |
| deflated Sharpe ratio (28 configs tried) | 0.85 |
| probability of backtest overfitting (CSCV) | 0.26 |
| its main config, with +2¢ slippage | +$4,516 |
It is positive in six of seven cities, and the configuration it chooses most often survives two cents of slippage and ten times the size. Most of the profit came in the last year, so this is not the 2023 edge that later decayed. The model's picks beat the mid by a wide margin, and costs take about half of that.
| component | dollars |
|---|---|
| alpha vs mid | +43,561 |
| spread paid | -12,434 |
| fees | -9,005 |
| net | +22,122 |
We tried 28 strategy configurations to get there, so the raw Sharpe overstates the result. The deflated Sharpe ratio, which corrects for the number of trials and for fat tails, came out at 0.85, and the probability of backtest overfitting at 0.26. Our bars were 0.95 and 0.20, so it cleared neither.
The sealed test
So we ran a test we could only run once. We wrote down the frozen strategy and the pass rule, pushed them to the repo, and only then scored July to October 2026, which nothing had touched. It passed: +$1,851 over 95 days, 80% of trades winning, a t of 1.77 against a bar of 1.645, and still positive with an extra cent of slippage. Trading noise with the same turnover lost $2,703 over the same days. The profit fell in each full month.
| month | dollars |
|---|---|
| July | +1,029 |
| August | +517 |
| September | +246 |
| October (3 days) | +60 |
The strategy passed a test it could not have been fit to, and its profit is shrinking about as fast as the 2023 edge did. Three months cannot tell us whether that is renewed decay, the August switch in Kalshi's settlement source, or noise.
We looked for causes. NBM's April 2026 upgrade did not make it visibly better at these stations. The market did improve, by about 0.15 nats in a year, and it now prices all of NBM. What is left of the GFS edge is seasonal, strong in summer and weak in winter, so part of the sealed test's monthly decline may be the season. On twelve cities the model never saw, it is level with the market, with no edge we can detect.
Daily lows
Kalshi started listing daily low temperatures in December 2025. With the same pipeline and family of strategies, the backtest cleared every bar we had set: a deflated Sharpe of 0.98, a probability of backtest overfitting of 0.06, and $13k in seven months. We sealed off September and October before looking, wrote down the rule, and scored it once. It made $36.
The model's probabilities still beat the market's on those weeks, by 0.019 nats with an interval clear of zero, but the profit was gone. The placebo suggests why. In the backtest even random trades made a little money, because a new market was badly calibrated. In the sealed weeks random trades lost. Most of the backtest profit looks like early mispricing that has since closed.
What improved the model, and what did not
We gave the model same-day inputs: five-minute station observations, NBM's forecast for the rest of the afternoon, and a new 2pm read. At 2pm the model beats the market by 0.017 nats over the last year, its biggest recent gain at any hour of the day. That did not become profit. By mid afternoon the order books are thin, with only 322 trades worth taking in three years, and resting orders were still picked off. Pulling our quotes whenever a new GFS run lands did not help either: the backtest almost never chose it.
We tried pretraining on synthetic data, as Jev does. We built 92,000 synthetic ladders from 55 weather stations Kalshi does not list, with a simulated market tuned until its spreads, overround and accuracy matched the real one. Pretraining doubled the model's edge when we gave it only a tenth of the real history, and made it slightly worse with all of it. It could help in a market that has just launched, but not in this one.
We tried a larger architecture: a transformer that lets every bucket of a ladder attend to every other, in place of the small per-bucket network, with the same inputs, training and pre-registered rules. At first size it landed within 0.004 nats of the simpler model. Eight times larger, it beat the MLP at two read times of four, by 0.010 nats the day before, where the rule asked for three. A learning curve suggests it is short of data. Retrained on a quarter, half and all of the history, the MLP gains 0.003 nats from four times the data, while the transformer gains 0.010 and goes from clearly worse to ahead. More of the same markets only arrives with time, and padding the training set with other targets did not help: adding the daily lows changed nothing, and adding twelve newer cities made it worse.
Most public signals we tested as inputs are already in the price. LAMP, the hourly NWS guidance that updates with the latest observations, estimates the final high 14 to 21% better than the NBM numbers the model was already using, yet adds nothing to it, presumably because traders watch LAMP too. Price momentum adds nothing either. Two inputs did help, and both take work to extract. NBM publishes its own spread in a text bulletin; letting the forecast's uncertainty follow it improves that forecast by 0.05 nats and nearly doubles isotherm's day-before edge. And which side has been crossing the spread in the last three hours, computed from the trade tape, adds about 0.005 nats before the day starts. An equal-weight average of the MLP, the large transformer and the spread model beats the MLP at all four read times, the first variant in the project to do that.
The largest gain came from more weather data. EMOS, the standard statistical correction, learns each city's forecast errors from a few years of that city alone. We downloaded the same two forecasts and the official highs for 576 weather stations across the country, 1.16 million station-days with no market attached, and trained one network on all of them. It beats the best per-city EMOS by 0.043 nats. Fed into the MLP, it more than doubles the edge over the market at 4pm the day before, from 0.014 to 0.032 nats, and it passed both gates we wrote down before it ran. Most of that gain comes from NBM's own spread, which the network also reads: compared with the spread model, the extra gain is too small to tell from zero.
Typed answers
We also scored the typed answers as someone asking isotherm questions would receive them. They never contradict each other, on any of 10,000 ladders. "Will it be above t" comes back with a third to two thirds less calibration error than the market's own odds, and an 80% range holds the high about as often as it claims, where the market's ranges are too wide. For "what will the high be", isotherm is no better than the market. So the market's estimate of the high is already good, and what it misprices is its uncertainty.
We then thought we had found the same thing in the prices alone: raise every bucket's price to a power of about 1.17 and the market beats itself by 0.010 nats. Testing it on S&P 500 and currency ladders showed it was a pricing artefact. Most buckets far from the action have no bid at all, only an ask of a few cents, and a mid-price counts that unbid ask as probability. Price each bucket at its bid and most of the gain disappears, on weather too. Against that stricter market, isotherm still wins at three of four read times, so its edge comes from the forecasts and not from how we priced the market.
Other markets
If the edge comes from the forecasts, it should also appear where useful public information takes skill to use. We tried two more of Kalshi's markets. Gas prices failed: a model built on wholesale futures was far worse than the market, because by the afternoon traders can watch real-time pump prices that are almost tomorrow's answer. Rain worked: calibrated NWS rain probabilities beat Kalshi's "will it rain tomorrow" markets across 25 cities, and on New York back to 2021, because the market overprices rain. The gain is small, mostly inside the bid-ask spread, and shrinking, as with temperature. The rain model is frozen and gets its own sealed test from 11 October.
We ran the same pipeline on MLB game winners, the other end of the liquidity spectrum: 4,484 games, Elo plus starting pitchers, one cent spreads and around a million contracts a game. The best blend landed within 0.001 nats of the market, so we found no edge there.
A recent paper on financial world models (Merchant et al., 2026) found that on a trillion stock-market observations, an untrained network with a fitted output layer is a strong baseline, and that results vary far more from month to month than from one random seed to the next. Both hold here. A transformer with its encoder frozen at random already beats the market and ties our trained MLP; training it adds only at 4pm the day before. Which month it is explains 97 to 99% of the variation in isotherm's monthly edge, and the seed explains under half a percent. That points to the forecasts as the source of the edge, and not learned representations.
To test that beyond weather, we started a small benchmark: 3,860 settled events across 13 Kalshi series, each priced 24, 6 and 1 hours before close. For the five with a free price feed (the S&P 500, Nasdaq 100, two currency pairs and oil) we asked whether a model built only from that feed beats the market even when the market is taken at its best price inside the bid-ask spread. For the S&P 500 and Nasdaq it clearly does not, and for currencies it ties, since those traders watch the same free price feed we do. Oil a day out is the one exception. Since we found it by looking, it is now frozen and gets its own sealed test on events from 12 October.
Before fetching a single price, we wrote down what should happen on Kalshi's other daily commodity ladders if the oil result was real: gold, silver, copper, Brent and natural gas. The prediction held. A day before close, the free futures price beats those books by 0.33 nats even at their best prices inside the spread. But the gain comes from a few ladders where the book had the eventual answer priced near zero, and a trader who took every mispriced quote would have made about nothing after fees. As with temperature, a measurably better forecast did not mean profit.
Where this stands
Everything we found by looking at old data now goes through a forward test. The MLP, the large transformer, the spread model, the order-flow model and the three-way ensemble were frozen on 6 October 2026, hashed, and committed before the first test read. The weather model joined a day later and is scored from 9 October. They get scored once on ladders through 5 April 2027.
The transformer needs more data, and the seven cities only add about 2,550 ladders a year. Kalshi now lists twelve more. Trained on all nineteen, each as a city of its own, the transformer gained where the MLP did not, in a first look at old data. That gets its own test on January to April 2027, a window that has not happened yet. Polymarket's weather markets looked like another cheap source, but they settle on hourly airport readings, which miss the day's true peak about a third of the time, so they resolve to a different number from Kalshi's.
The strategy now runs live in shadow mode. Every day at 4pm local, a scheduled job scores the next day's ladders and commits its paper trades before the outcome exists, so the commit timestamps show nothing was backfilled. A decay alarm tracks a rolling 30-day score against the market and flips to STOP if two windows in a row sit clearly below zero. If the alarm stays quiet for a month, the next step is real execution. If it fires, we will write that up too.
The practices we would keep on any project like this: score against the market and not against zero; use proper scoring rules for both training and evaluation; keep a control that can only win if the pipeline leaks; count your trials and deflate your Sharpe; distrust backtests on new markets; and seal some data before you start.
isotherm is live at isotherm.onrender.com, as a plain API and as an MCP server that an agent like Claude can call. "What's the chance Chicago hits 75 tomorrow" comes back as one calibrated number, read off the same distribution as every other answer. It is on free hosting, so the first question after a quiet spell takes up to a minute.
Results are on the isotherm dashboard. Every number here is reproducible from physicalreasoning/isotherm, with the code, raw results and the full findings record, including the bugs we caught and the gate we had to amend. The live paper ledger is on the shadow-ledger branch.
References. G. W. Brier, Verification of forecasts expressed in terms of probability, Monthly Weather Review, 1950. T. Gneiting, A. E. Raftery, A. H. Westveld, T. Goldman, Calibrated probabilistic forecasting using ensemble model output statistics, Monthly Weather Review, 2005. T. Gneiting, A. E. Raftery, Strictly proper scoring rules, prediction, and estimation, JASA, 2007. H. R. Glahn, D. A. Lowry, The use of model output statistics in objective weather forecasting, Journal of Applied Meteorology, 1972. J. P. Craven, D. E. Rudack, P. E. Shafer, National Blend of Models, Journal of Operational Meteorology, 2020. J. L. Kelly, A new interpretation of information rate, Bell System Technical Journal, 1956. L. R. Glosten, P. R. Milgrom, Bid, ask and transaction prices in a specialist market with heterogeneously informed traders, Journal of Financial Economics, 1985. D. H. Bailey, M. López de Prado, The deflated Sharpe ratio, Journal of Portfolio Management, 2014. D. H. Bailey, J. M. Borwein, M. López de Prado, Q. J. Zhu, The probability of backtest overfitting, Journal of Computational Finance, 2017. The typed question design follows TypeSafe AI's Jev; isotherm is an independent open implementation, not affiliated with TypeSafe. Full list in the repository. Code archived as doi:10.5281/zenodo.23167633.