Limits: what these figures show, and what they do not

Figures as of 14 November 2026. Version française.

A sample that is still small

29 questions are resolved (16 Yes, 13 No; 28 independent after grouping questions that rest on the same event). The overall Brier score is 0.172, against 0.250 for the "50% for everything" reference. With 29 resolutions the uncertainty remains large: a single big miss moves this figure by several hundredths. This is not proof of skill.

Where the score is solid, and where it is less so

• On 6 questions where a base rate could be computed (for example the FDA registry or Fed futures), the Brier is 0.038 against 0.056 for that rate alone (skill score 0.32). 6 cases allow no conclusion.
• On 23 questions where the base rate is only estimated, the Brier is 0.209 against 0.250 for the naive reference, with no skill score.
• By domain: Finance, 24 of 29 questions (Brier 0.196). Too few questions to interpret: Medicine (4 questions, 0.065), AI and tech (1 question, 0.048).
• Excluding medicine: 25 questions (24 independent), Brier 0.190.

What we do not do

• No published estimate is recomputed after the fact.
• The estimates do not correct themselves: the models do not learn from past results. Every method change is decided by hand, dated and published.
• We claim no statistically established edge until 30 independent resolutions are reached, with a confidence interval that excludes zero.

Other limits

AI models can be wrong. Part of the verdicts is checked by hand. Some neighbouring questions share the same event. The track record is short.