Can a local open-weights model stand in for the hosted decision model on the prediction-market news radar?
On the same headlines, markets, questions and decision code, does a local open-weights model reach the same decisions as the hosted model the radar uses today often enough to replace it?
No. The radar logged 30,868 headline-market pairs from 4 to 22 November 2026; 17,284 could be replayed, and the verdict rests on the 16,919 from Kalshi and Polymarket (5,063 headlines; 1,690 more Kalshi pairs had no price to rebuild and Polymarket US is reported apart). On those the local model's decisions agreed with the hosted model's barely more than chance: Cohen's kappa 0.036 (95% interval 0.019 to 0.054) against a bar of 0.60. On the 132 pairs both called, the local model named NO every time, so the 49% side agreement (interval 41% to 57%; bar 85%) is just the hosted model's share of NO calls, and the local model called about three times as many markets (ratio 2.97, interval 2.62 to 3.39; allowed 0.67 to 1.5). By the pre-registered rule the replay verdict is NOT REPLACEABLE: kappa's upper bound is under 0.40 and the call-rate interval lies wholly outside 0.5 to 2. The models disagree most on whether a headline bears on a market at all (rank correlation -0.13) and agree best on whether it reports a new fact (0.47). The 4-week forward part is still running and cannot change a NOT REPLACEABLE replay into a switch. The study measures agreement only; it does not say which model is right.
Backtested by QS Research
Pre-registration
Fixed before any answer of the local model on these inputs existed: every verdict row the radar logged from 4 November 2026 that has a price at its decision time is replayed through the local model with the radar's own four questions and its own decision code, unchanged. Primary measures are decision agreement (a call against no call; a call is a signal or a lean, and for the hosted model also a call the radar's 6-hour repeat clock relabelled) with Cohen's kappa, side agreement among pairs both call, per-question rank correlation and kappa, and the ratio of call rates, each with a 95% percentile interval from 4,000 resamples of whole headlines, seed 13. REPLACEABLE needs kappa's lower bound at 0.60 or more, side agreement's lower bound at 0.85 or more and the call-rate ratio's interval inside 0.67 to 1.5; NOT REPLACEABLE is kappa's upper bound under 0.40 or the ratio's interval wholly outside 0.5 to 2; otherwise INCONCLUSIVE; no verdict under 100 headlines with a call. The replay gives the verdict and a 4-week forward part must confirm before any switch. Outcomes are secondary, on two venues only, and never decisive.
Registered 2026-11-21Amendments
- 2026-11-21 Written before the first answer of the local model: the model revision; where market text, news summaries and prices come from; the verdict is computed on Kalshi and Polymarket, where a price could be rebuilt for all but 1,690 Kalshi pairs, and not on Polymarket US, where only pairs the hosted model scored relevant have a price; 1,906 rows from a parallel run on an old machine are left out; how relabelled repeat calls are classed; per-question thresholds; bootstrap details; a truncation breakdown. approved: Recorded by the running agent before the data it governs was read; reported to the study owner, 2026-11-22.
Trials · 0 trials · 0 passed · judged at N = 1 trials; 2972 trials run in total
| Trial | Verdict | |
|---|---|---|
| Kalshi and Polymarket together (the verdict's rows) | RESULT | not counted |
| kalshi | RESULT | not counted |
| polymarket | RESULT | not counted |
| polymarket_us (priced rows only: selected on the hosted model's relevance; not in the verdict) | RESULT | not counted |
| input-exact rows, every venue | RESULT | not counted |
| Kalshi and Polymarket rows the local model did not cut (descriptive) | RESULT | not counted |
| Kalshi and Polymarket rows the local model cut (descriptive) | RESULT | not counted |
| Kalshi and Polymarket together (the verdict's rows) · relevant | RESULT | not counted |
| Kalshi and Polymarket together (the verdict's rows) · move | RESULT | not counted |
| Kalshi and Polymarket together (the verdict's rows) · new_fact | RESULT | not counted |
| Kalshi and Polymarket together (the verdict's rows) · priced_in | RESULT | not counted |
| kalshi · relevant | RESULT | not counted |
| kalshi · move | RESULT | not counted |
| kalshi · new_fact | RESULT | not counted |
| kalshi · priced_in | RESULT | not counted |
| polymarket · relevant | RESULT | not counted |
| polymarket · move | RESULT | not counted |
| polymarket · new_fact | RESULT | not counted |
| polymarket · priced_in | RESULT | not counted |
| polymarket_us (priced rows only: selected on the hosted model's relevance; not in the verdict) · relevant | RESULT | not counted |
| polymarket_us (priced rows only: selected on the hosted model's relevance; not in the verdict) · move | RESULT | not counted |
| polymarket_us (priced rows only: selected on the hosted model's relevance; not in the verdict) · new_fact | RESULT | not counted |
| polymarket_us (priced rows only: selected on the hosted model's relevance; not in the verdict) · priced_in | RESULT | not counted |
| kalshi | RESULT | not counted |
| polymarket | RESULT | not counted |
| polymarket_us | RESULT | not counted |
| Hosted model's calls | RESULT | not counted |
| Local model's calls | RESULT | not counted |
| Local minus hosted, union of calls | RESULT | not counted |
| Hosted model's calls | RESULT | not counted |
| Local model's calls | RESULT | not counted |
| Local minus hosted, union of calls | RESULT | not counted |
| polymarket | RESULT | not counted |
| polymarket_us | RESULT | not counted |
Limitations
- The local model did not see exactly what the hosted model saw: most headlines came from feeds whose summary was never logged, so it got none; the market text is today's, not the decision-time text; and for most pairs the price was rebuilt from the venue's price history. Only 2 pairs had exact inputs. These differences count against the local model.
- 1,690 of 18,539 Kalshi pairs had no price in the 10 minutes before the decision and were left out; all were pairs the hosted model scored not relevant.
- Polymarket US keeps no price history, so only the 365 pairs the hosted model scored relevant could be replayed there. That venue is reported beside the verdict and is not part of it; this was decided by amendment before the first answer.
- 228 of the hosted model's 698 calls in the verdict's pairs are calls the radar's repeat clock relabelled; they count as calls, as the amendment fixed before the first answer.
- Agreement is not accuracy: where the two differ, this study does not say which is right.
- The outcome comparison covers 61 settled calls on one venue and cannot tell the two models apart.
- The local model reads at most 512 tokens per question; on 30% of Polymarket pairs and 4% of Polymarket US pairs the end of the input, where the price sits, was cut for one question. No Kalshi pair was cut.
- No signed engine release and no pinned data snapshot: the input files and the code that produced them are recorded by content hash.
- The forward part has no rows yet.
The numbers: every trial's checks, metrics, curve and yearly results, and the provenance
2 published versions · current 05a8ebe88d65a353, published 2026-11-22