QSQuantSignals Research/Studies
Try FST free

← All studies

FAIL

Can a local open-weights model stand in for the hosted decision model on the prediction-market news radar?

On the same headlines, markets, questions and decision code, does a local open-weights model reach the same decisions as the hosted model the radar uses today often enough to replace it?

No. The radar logged 30,868 headline-market pairs from 4 to 22 November 2026; 17,284 could be replayed, and the verdict rests on the 16,919 from Kalshi and Polymarket (5,063 headlines; 1,690 more Kalshi pairs had no price to rebuild and Polymarket US is reported apart). On those the local model's decisions agreed with the hosted model's barely more than chance: Cohen's kappa 0.036 (95% interval 0.019 to 0.054) against a bar of 0.60. On the 132 pairs both called, the local model named NO every time, so the 49% side agreement (interval 41% to 57%; bar 85%) is just the hosted model's share of NO calls, and the local model called about three times as many markets (ratio 2.97, interval 2.62 to 3.39; allowed 0.67 to 1.5). By the pre-registered rule the replay verdict is NOT REPLACEABLE: kappa's upper bound is under 0.40 and the call-rate interval lies wholly outside 0.5 to 2. The models disagree most on whether a headline bears on a market at all (rank correlation -0.13) and agree best on whether it reports a new fact (0.47). The 4-week forward part is still running and cannot change a NOT REPLACEABLE replay into a switch. The study measures agreement only; it does not say which model is right.

Backtested by QS Research

Pre-registration

Fixed before any answer of the local model on these inputs existed: every verdict row the radar logged from 4 November 2026 that has a price at its decision time is replayed through the local model with the radar's own four questions and its own decision code, unchanged. Primary measures are decision agreement (a call against no call; a call is a signal or a lean, and for the hosted model also a call the radar's 6-hour repeat clock relabelled) with Cohen's kappa, side agreement among pairs both call, per-question rank correlation and kappa, and the ratio of call rates, each with a 95% percentile interval from 4,000 resamples of whole headlines, seed 13. REPLACEABLE needs kappa's lower bound at 0.60 or more, side agreement's lower bound at 0.85 or more and the call-rate ratio's interval inside 0.67 to 1.5; NOT REPLACEABLE is kappa's upper bound under 0.40 or the ratio's interval wholly outside 0.5 to 2; otherwise INCONCLUSIVE; no verdict under 100 headlines with a call. The replay gives the verdict and a 4-week forward part must confirm before any switch. Outcomes are secondary, on two venues only, and never decisive.

Registered 2026-11-21

Amendments

Trials · 0 trials · 0 passed · judged at N = 1 trials; 2972 trials run in total

TrialVerdict
Kalshi and Polymarket together (the verdict's rows)RESULTnot counted
kalshiRESULTnot counted
polymarketRESULTnot counted
polymarket_us (priced rows only: selected on the hosted model's relevance; not in the verdict)RESULTnot counted
input-exact rows, every venueRESULTnot counted
Kalshi and Polymarket rows the local model did not cut (descriptive)RESULTnot counted
Kalshi and Polymarket rows the local model cut (descriptive)RESULTnot counted
Kalshi and Polymarket together (the verdict's rows) · relevantRESULTnot counted
Kalshi and Polymarket together (the verdict's rows) · moveRESULTnot counted
Kalshi and Polymarket together (the verdict's rows) · new_factRESULTnot counted
Kalshi and Polymarket together (the verdict's rows) · priced_inRESULTnot counted
kalshi · relevantRESULTnot counted
kalshi · moveRESULTnot counted
kalshi · new_factRESULTnot counted
kalshi · priced_inRESULTnot counted
polymarket · relevantRESULTnot counted
polymarket · moveRESULTnot counted
polymarket · new_factRESULTnot counted
polymarket · priced_inRESULTnot counted
polymarket_us (priced rows only: selected on the hosted model's relevance; not in the verdict) · relevantRESULTnot counted
polymarket_us (priced rows only: selected on the hosted model's relevance; not in the verdict) · moveRESULTnot counted
polymarket_us (priced rows only: selected on the hosted model's relevance; not in the verdict) · new_factRESULTnot counted
polymarket_us (priced rows only: selected on the hosted model's relevance; not in the verdict) · priced_inRESULTnot counted
kalshiRESULTnot counted
polymarketRESULTnot counted
polymarket_usRESULTnot counted
Hosted model's callsRESULTnot counted
Local model's callsRESULTnot counted
Local minus hosted, union of callsRESULTnot counted
Hosted model's callsRESULTnot counted
Local model's callsRESULTnot counted
Local minus hosted, union of callsRESULTnot counted
polymarketRESULTnot counted
polymarket_usRESULTnot counted

Limitations

The numbers: every trial's checks, metrics, curve and yearly results, and the provenance

2 published versions · current 05a8ebe88d65a353, published 2026-11-22

Put these signals to work

This page is free. Members get the live version of every QS Research signal. FST, the QuantSignals trading agent, uses QS Research as its brain.

Join freeMeet FSTSee plans