Open problems: starter projects in world-model evaluation
Starter projects, from a day to a few weeks each, drawn from gaps our own work left open. Each links to a GitHub issue, where the details live and where you can claim it. Some will end in a negative result, and we publish those too. New here? Read Start here first.
Forecasting against the market (isotherm)
A third forecast source for the station weather model
Does adding ECMWF's AIFS or IFS improve isotherm's station weather model at the 16:00 day-before read? issue #58
An intra-hour benchmark for Bitcoin ladders
Does free outside data carry information beyond the quote on Kalshi's hourly Bitcoin ladders, read inside the hour before close? issue #59
A commodity model that holds up an hour before close
Can a better outside model make free futures prices competitive with Kalshi's commodity ladders 6 hours and 1 hour before close? issue #60
Score the live API's answers
How good are the answers the live API actually serves? issue #61
Sealed forward tests (forward)
Spread-aware weather entries
Would EMOS with NBM's own spread, or isotherm's station network, beat forward's best weather entry? issue #3
Does fine-tuning help a time-series foundation model beat copying?
Zero-shot, no released time-series foundation model beats copying the last price on Kalshi's price ladders. Does one beat it after fine-tuning on the underlyings? issue #4
Reproduce the price-ladder dry run from forward alone
Can a new contributor reproduce task 2 from a clone of forward? issue #2
Probing learned representations (causal-jepa)
Finish the convergence run that could reopen the programme
Does more training make the representation worse (finding 8), or was that an artefact of wide readouts? issue #4
Does the one surviving lift replicate at ten minutes?
Does the training lift on future realised volatility survive the raw-feature control at a ten-minute horizon, as it does at five? issue #5
Headroom on a different asset class, before any training
Can the probe targets that failed on hourly crypto ladders rank representations on another asset class? issue #6
Run the experiments on a locally rebuilt corpus
Can a contributor rebuild the corpus and run every script without extra configuration? issue #7
Evaluating under intervention
Design an intervention test for world models
Can we build a small, reproducible test of whether a model's predicted outcome of an action matches what actually happens? issue #5
Claim a problem by commenting on its issue. Propose a new one by opening an issue on the repository it belongs to. Contributing guides: in each repository, isotherm, forward, causal-jepa. Background: Start here.