Examenos

Four equally weighted evaluations: AA-Omniscience-Decision, HLE-MultiChoice (513 text-only questions), When2Call and WildGuardTest.

Exact named AA decision system; index v1.0. Served precision and detailed execution date not published in inspected chart.

Rounded chart label manually verified in source; not the ordinary language-model Intelligence Index.

Independent AA results. Omniscience splits its 25% between accuracy and non-hallucination. Displayed integer chart values, not undisclosed precise values. No automated AA import or refresh adapter.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-17 · Revision: Manually checked chart · 2026-10-17 · Published observations; result reuse terms not specified

BFCL formulated as typed decisions; published case exact accuracy for this board population.

Board identity clef-omni; evaluator run. BF16 joint-schema path, raised 65,536-token budget; 33 overlong ToolRet requests count wrong.

Raw 0.9864; {"requests": 1694, "answered": 1694, "unsupported": 0, "errors": 0}; coverage/error rules are source-defined.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: 1694

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

SGD/SGD-X formulated as typed decisions; published macro-F1 for this board population.

Board identity clef-omni; evaluator run. BF16 joint-schema path, raised 65,536-token budget; 33 overlong ToolRet requests count wrong.

Raw 0.4306; {"requests": 2500, "answered": 2500, "unsupported": 0, "errors": 0}; coverage/error rules are source-defined.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: 2500

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

ContractNLI formulated as typed decisions; published macro-F1 for this board population.

Board identity clef-omni; evaluator run. BF16 joint-schema path, raised 65,536-token budget; 33 overlong ToolRet requests count wrong.

Raw 0.8237; {"requests": 123, "answered": 123, "unsupported": 0, "errors": 0}; coverage/error rules are source-defined.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: 123

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

ANLI formulated as typed decisions; published macro-F1 for this board population.

Board identity clef-omni; evaluator run. BF16 joint-schema path, raised 65,536-token budget; 33 overlong ToolRet requests count wrong.

Raw 0.6363; {"requests": 3200, "answered": 3200, "unsupported": 0, "errors": 0}; coverage/error rules are source-defined.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: 3200

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

ToolRet formulated as typed decisions; published nDCG@10 for this board population.

Board identity clef-omni; evaluator run. BF16 joint-schema path, raised 65,536-token budget; 33 overlong ToolRet requests count wrong.

Raw 0.6524; {"requests": 685, "answered": 685, "unsupported": 0, "errors": 0}; coverage/error rules are source-defined.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: 685

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

BPoMP formulated as typed decisions; published accuracy for this board population.

Board identity clef-omni; evaluator run. BF16 joint-schema path, raised 65,536-token budget; 33 overlong ToolRet requests count wrong.

Raw 0.9728; {"requests": 5000, "answered": 5000, "unsupported": 0, "errors": 0}; coverage/error rules are source-defined.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: 5000

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

Humicroedit formulated as typed decisions; published accuracy for this board population.

Board identity clef-omni; evaluator run. BF16 joint-schema path, raised 65,536-token budget; 33 overlong ToolRet requests count wrong.

Raw 0.7108; {"requests": 2628, "answered": 2628, "unsupported": 0, "errors": 0}; coverage/error rules are source-defined.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: 2628

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

POP909-CL formulated as typed decisions; published accuracy for this board population.

Board identity clef-omni; evaluator run. BF16 joint-schema path, raised 65,536-token budget; 33 overlong ToolRet requests count wrong.

Raw 0.022; {"requests": 2000, "answered": 2000, "unsupported": 0, "errors": 0}; coverage/error rules are source-defined.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: 2000

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

cfcolor formulated as typed decisions; published accuracy for this board population.

Board identity clef-omni; evaluator run. BF16 joint-schema path, raised 65,536-token budget; 33 overlong ToolRet requests count wrong.

Raw 0.6324; {"requests": 5000, "answered": 5000, "unsupported": 0, "errors": 0}; coverage/error rules are source-defined.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: 5000

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

MMLU formulated as typed decisions; published accuracy for this board population.

Board identity clef-omni; evaluator run. BF16 joint-schema path, raised 65,536-token budget; 33 overlong ToolRet requests count wrong.

Raw 0.9235; {"requests": 14033, "answered": 14033, "unsupported": 0, "errors": 0}; coverage/error rules are source-defined.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: 14033

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

GPQA Diamond formulated as typed decisions; published accuracy for this board population.

Board identity clef-omni; evaluator run. BF16 joint-schema path, raised 65,536-token budget; 33 overlong ToolRet requests count wrong.

Raw 0.4694; {"requests": 196, "answered": 196, "unsupported": 0, "errors": 0}; coverage/error rules are source-defined.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: 196

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

ARC-Easy formulated as typed decisions; published accuracy for this board population.

Board identity clef-omni; evaluator run. BF16 joint-schema path, raised 65,536-token budget; 33 overlong ToolRet requests count wrong.

Raw 0.9945; {"requests": 2376, "answered": 2376, "unsupported": 0, "errors": 0}; coverage/error rules are source-defined.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: 2376

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

ARC-Challenge formulated as typed decisions; published accuracy for this board population.

Board identity clef-omni; evaluator run. BF16 joint-schema path, raised 65,536-token budget; 33 overlong ToolRet requests count wrong.

Raw 0.9787; {"requests": 1172, "answered": 1172, "unsupported": 0, "errors": 0}; coverage/error rules are source-defined.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: 1172

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

WinoGrande formulated as typed decisions; published accuracy for this board population.

Board identity clef-omni; evaluator run. BF16 joint-schema path, raised 65,536-token budget; 33 overlong ToolRet requests count wrong.

Raw 0.9676; {"requests": 1267, "answered": 1267, "unsupported": 0, "errors": 0}; coverage/error rules are source-defined.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: 1267

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

HellaSwag formulated as typed decisions; published accuracy for this board population.

Board identity clef-omni; evaluator run. BF16 joint-schema path, raised 65,536-token budget; 33 overlong ToolRet requests count wrong.

Raw 0.9895; {"requests": 10042, "answered": 10042, "unsupported": 0, "errors": 0}; coverage/error rules are source-defined.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: 10042

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

API-Bank formulated as typed decisions; published accuracy for this board population.

Board identity clef-omni; evaluator run. BF16 joint-schema path, raised 65,536-token budget; 33 overlong ToolRet requests count wrong.

Raw 0.8701; {"requests": 508, "answered": 508, "unsupported": 0, "errors": 0}; coverage/error rules are source-defined.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: 508

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

GSM8K formulated as typed decisions; published accuracy for this board population.

Board identity clef-omni; evaluator run. BF16 joint-schema path, raised 65,536-token budget; 33 overlong ToolRet requests count wrong.

Raw 0.6785; {"requests": 2638, "answered": 2638, "unsupported": 0, "errors": 0}; coverage/error rules are source-defined.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: 2638

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

ChessBench formulated as typed decisions; published accuracy for this board population.

Board identity clef-omni; evaluator run. BF16 joint-schema path, raised 65,536-token budget; 33 overlong ToolRet requests count wrong.

Raw 0.2192; {"requests": 5000, "answered": 5000, "unsupported": 0, "errors": 0}; coverage/error rules are source-defined.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: 5000

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

MuSR formulated as typed decisions; published accuracy for this board population.

Board identity clef-omni; evaluator run. BF16 joint-schema path, raised 65,536-token budget; 33 overlong ToolRet requests count wrong.

Raw 0.8431; {"requests": 752, "answered": 752, "unsupported": 0, "errors": 0}; coverage/error rules are source-defined.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: 752

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

SATA-Bench formulated as typed decisions; published case exact accuracy for this board population.

Board identity clef-omni; evaluator run. BF16 joint-schema path, raised 65,536-token budget; 33 overlong ToolRet requests count wrong.

Raw 0.3133; {"requests": 1650, "answered": 1650, "unsupported": 0, "errors": 0}; coverage/error rules are source-defined.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: 1650

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

BRIGHT formulated as typed decisions; published nDCG@10 for this board population.

Board identity clef-omni; evaluator run. BF16 joint-schema path, raised 65,536-token budget; 33 overlong ToolRet requests count wrong.

Raw 0.4422; {"requests": 220, "answered": 220, "unsupported": 0, "errors": 0}; coverage/error rules are source-defined.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: 220

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

Amazon ESCI formulated as typed decisions; published macro-F1 for this board population.

Board identity clef-omni; evaluator run. BF16 joint-schema path, raised 65,536-token budget; 33 overlong ToolRet requests count wrong.

Raw 0.5784; {"requests": 5000, "answered": 5000, "unsupported": 0, "errors": 0}; coverage/error rules are source-defined.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: 5000

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

ACOS formulated as typed decisions; published per-review F1 for this board population.

Board identity clef-omni; evaluator run. BF16 joint-schema path, raised 65,536-token budget; 33 overlong ToolRet requests count wrong.

Raw 0.242; {"requests": 1565, "answered": 1565, "unsupported": 0, "errors": 0}; coverage/error rules are source-defined.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: 1565

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

FinEntity formulated as typed decisions; published macro-F1 for this board population.

Board identity clef-omni; evaluator run. BF16 joint-schema path, raised 65,536-token budget; 33 overlong ToolRet requests count wrong.

Raw 0.9665; {"requests": 979, "answered": 979, "unsupported": 0, "errors": 0}; coverage/error rules are source-defined.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: 979

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

BANKING77 formulated as typed decisions; published macro-F1 for this board population.

Board identity clef-omni; evaluator run. BF16 joint-schema path, raised 65,536-token budget; 33 overlong ToolRet requests count wrong.

Raw 0.9423; {"requests": 3080, "answered": 3080, "unsupported": 0, "errors": 0}; coverage/error rules are source-defined.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: 3080

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

iSarcasmEval formulated as typed decisions; published Sarcasm F1 · track A, English for this board population.

Board identity clef-omni; evaluator run. BF16 joint-schema path, raised 65,536-token budget; 33 overlong ToolRet requests count wrong.

Raw 0.3725; {"requests": 4600, "answered": 4600, "unsupported": 0, "errors": 0}; coverage/error rules are source-defined.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: 4600

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

VAST formulated as typed decisions; published macro-F1 for this board population.

Board identity clef-omni; evaluator run. BF16 joint-schema path, raised 65,536-token budget; 33 overlong ToolRet requests count wrong.

Raw 0.5145; {"requests": 3006, "answered": 3006, "unsupported": 0, "errors": 0}; coverage/error rules are source-defined.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: 3006

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

NLI4CT formulated as typed decisions; published macro-F1 for this board population.

Board identity clef-omni; evaluator run. BF16 joint-schema path, raised 65,536-token budget; 33 overlong ToolRet requests count wrong.

Raw 0.779; {"requests": 5500, "answered": 5500, "unsupported": 0, "errors": 0}; coverage/error rules are source-defined.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: 5500

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

CRUXEval formulated as typed decisions; published accuracy for this board population.

Board identity clef-omni; evaluator run. BF16 joint-schema path, raised 65,536-token budget; 33 overlong ToolRet requests count wrong.

Raw 0.8544; {"requests": 570, "answered": 570, "unsupported": 0, "errors": 0}; coverage/error rules are source-defined.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: 570

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

CLadder formulated as typed decisions; published accuracy for this board population.

Board identity clef-omni; evaluator run. BF16 joint-schema path, raised 65,536-token budget; 33 overlong ToolRet requests count wrong.

Raw 0.972; {"requests": 5000, "answered": 5000, "unsupported": 0, "errors": 0}; coverage/error rules are source-defined.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: 5000

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

HLE formulated as typed decisions; published accuracy for this board population.

Board identity clef-omni; evaluator run. BF16 joint-schema path, raised 65,536-token budget; 33 overlong ToolRet requests count wrong.

Raw 0.0918; {"requests": 501, "answered": 501, "unsupported": 0, "errors": 0}; coverage/error rules are source-defined.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: 501

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

CLINC150+OOS formulated as typed decisions; published macro-F1 for this board population.

Board identity clef-omni; evaluator run. BF16 joint-schema path, raised 65,536-token budget; 33 overlong ToolRet requests count wrong.

Raw 0.9751; {"requests": 5500, "answered": 5500, "unsupported": 0, "errors": 0}; coverage/error rules are source-defined.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: 5500

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

Habermas Machine formulated as typed decisions; published accuracy for this board population.

Board identity clef-omni; evaluator run. BF16 joint-schema path, raised 65,536-token budget; 33 overlong ToolRet requests count wrong.

Raw 0.6999; {"requests": 1676, "answered": 1676, "unsupported": 0, "errors": 0}; coverage/error rules are source-defined.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: 1676

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

PhishNChips phishing decisions formulated as typed decisions; published accuracy for this board population.

Board identity clef-omni; evaluator run. BF16 joint-schema path, raised 65,536-token budget; 33 overlong ToolRet requests count wrong.

Raw 0.7245; {"requests": 2000, "answered": 2000, "unsupported": 0, "errors": 0}; coverage/error rules are source-defined.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: 2000

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

MMLU-Pro formulated as typed decisions; published accuracy for this board population.

Board identity clef-omni; evaluator run. BF16 joint-schema path, raised 65,536-token budget; 33 overlong ToolRet requests count wrong.

Raw 0.6927; {"requests": 12032, "answered": 12032, "unsupported": 0, "errors": 0}; coverage/error rules are source-defined.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: 12032

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

BBH formulated as typed decisions; published accuracy for this board population.

Board identity clef-omni; evaluator run. BF16 joint-schema path, raised 65,536-token budget; 33 overlong ToolRet requests count wrong.

Raw 0.6895; {"requests": 5507, "answered": 5507, "unsupported": 0, "errors": 0}; coverage/error rules are source-defined.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: 5507

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

RAGTruth formulated as typed decisions; published F1 on hallucinated class for this board population.

Board identity clef-omni; evaluator run. BF16 joint-schema path, raised 65,536-token budget; 33 overlong ToolRet requests count wrong.

Raw 0.3451; {"requests": 2700, "answered": 2700, "unsupported": 0, "errors": 0}; coverage/error rules are source-defined.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: 2700

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

RouterBench formulated as typed decisions; published selected quality (quality objective) for this board population.

Board identity clef-omni; evaluator run. BF16 joint-schema path, raised 65,536-token budget; 33 overlong ToolRet requests count wrong.

Raw 0.7996; {"requests": 10000, "answered": 10000, "unsupported": 0, "errors": 0}; coverage/error rules are source-defined.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: 10000

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

HoVer formulated as typed decisions; published accuracy for this board population.

Board identity clef-omni; evaluator run. BF16 joint-schema path, raised 65,536-token budget; 33 overlong ToolRet requests count wrong.

Raw 0.6245; {"requests": 4000, "answered": 4000, "unsupported": 0, "errors": 0}; coverage/error rules are source-defined.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: 4000

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

When2Call formulated as typed decisions; published accuracy for this board population.

Board identity clef-omni; evaluator run. BF16 joint-schema path, raised 65,536-token budget; 33 overlong ToolRet requests count wrong.

Raw 0.5665; {"requests": 3652, "answered": 3652, "unsupported": 0, "errors": 0}; coverage/error rules are source-defined.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: 3652

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

New Yorker formulated as typed decisions; published accuracy for this board population.

Board identity clef-omni; evaluator run. BF16 joint-schema path, raised 65,536-token budget; 33 overlong ToolRet requests count wrong.

Raw 0.6648; {"requests": 528, "answered": 528, "unsupported": 0, "errors": 0}; coverage/error rules are source-defined.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: 528

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

Home appliance simulator formulated as typed decisions; published case exact accuracy for this board population.

Board identity clef-omni; evaluator run. BF16 joint-schema path, raised 65,536-token budget; 33 overlong ToolRet requests count wrong.

Raw 0.6818; {"requests": 88, "answered": 88, "unsupported": 0, "errors": 0}; coverage/error rules are source-defined.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: 88

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

Calibration on the source-defined text sample; confidence alone is not a quality ranking.

Board identity clef-omni; evaluator run. BF16 joint-schema path, raised 65,536-token budget; 33 overlong ToolRet requests count wrong.

Raw acc: 0.7715; 31 benchmarks. ECE uses the publisher bins and weighting; native Brier convention stays distinct from other studies.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported. Uses roughly one sixth of requests with discrete correct answers; benchmarks receive equal weight. ToolRet, BRIGHT and RouterBench are excluded. Known training exposure is penalized.

Checked / reported: 2026-10-17 · Samples: 34411

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

Calibration on the source-defined text sample; confidence alone is not a quality ranking.

Board identity clef-omni; evaluator run. BF16 joint-schema path, raised 65,536-token budget; 33 overlong ToolRet requests count wrong.

Raw brier: 0.3234; 31 benchmarks. ECE uses the publisher bins and weighting; native Brier convention stays distinct from other studies.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported. Uses roughly one sixth of requests with discrete correct answers; benchmarks receive equal weight. ToolRet, BRIGHT and RouterBench are excluded. Known training exposure is penalized.

Checked / reported: 2026-10-17 · Samples: 34411

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

Calibration on the source-defined text sample; confidence alone is not a quality ranking.

Board identity clef-omni; evaluator run. BF16 joint-schema path, raised 65,536-token budget; 33 overlong ToolRet requests count wrong.

Raw conf: 0.7844; 31 benchmarks. ECE uses the publisher bins and weighting; native Brier convention stays distinct from other studies.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported. Uses roughly one sixth of requests with discrete correct answers; benchmarks receive equal weight. ToolRet, BRIGHT and RouterBench are excluded. Known training exposure is penalized.

Checked / reported: 2026-10-17 · Samples: 34411

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

Calibration on the source-defined text sample; confidence alone is not a quality ranking.

Board identity clef-omni; evaluator run. BF16 joint-schema path, raised 65,536-token budget; 33 overlong ToolRet requests count wrong.

Raw ece: 0.0403; 31 benchmarks. ECE uses the publisher bins and weighting; native Brier convention stays distinct from other studies.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported. Uses roughly one sixth of requests with discrete correct answers; benchmarks receive equal weight. ToolRet, BRIGHT and RouterBench are excluded. Known training exposure is penalized.

Checked / reported: 2026-10-17 · Samples: 34411

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

Published Median latency for the board setup; hosted network and local inference are separate.

Board identity clef-omni; evaluator run. BF16 joint-schema path, raised 65,536-token budget; 33 overlong ToolRet requests count wrong.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported. Local GPU/runtime can differ between submitted and evaluator runs; not isolated architecture speed.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

Published p95 latency for the board setup; hosted network and local inference are separate.

Board identity clef-omni; evaluator run. BF16 joint-schema path, raised 65,536-token budget; 33 overlong ToolRet requests count wrong.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported. Local GPU/runtime can differ between submitted and evaluator runs; not isolated architecture speed.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

Text Decision Index 0.3.2 · Private new-domain score

Board identity clef-omni; evaluator run. BF16 joint-schema path, raised 65,536-token budget; 33 overlong ToolRet requests count wrong.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

Text Decision Index 0.3.2 · Public score

Board identity clef-omni; evaluator run. BF16 joint-schema path, raised 65,536-token budget; 33 overlong ToolRet requests count wrong.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

Text Decision Index 0.3.2 · Private same-skill score

Board identity clef-omni; evaluator run. BF16 joint-schema path, raised 65,536-token budget; 33 overlong ToolRet requests count wrong.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

Text Decision Index 0.3.2 · Full score

Board identity clef-omni; evaluator run. BF16 joint-schema path, raised 65,536-token budget; 33 overlong ToolRet requests count wrong.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

Typed vision decisions on BLINK; private population.

Vision board identity clef-omni. Effective image budget 1.6 MP.

Vision board 0.3.2, 50% public and 50% private; public/private are distinct from the text board. Source benchmark weights and image caps apply. author cap 1,048,576 px lifted to the board cap via the documented media_kwargs (processor max 12.8 MP) Private normalized scores may be below zero; no clipping. Public sample count: 961.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-16T19:00:13Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

Typed vision decisions on BLINK; pub population.

Vision board identity clef-omni. Effective image budget 1.6 MP.

Vision board 0.3.2, 50% public and 50% private; public/private are distinct from the text board. Source benchmark weights and image caps apply. author cap 1,048,576 px lifted to the board cap via the documented media_kwargs (processor max 12.8 MP) Private normalized scores may be below zero; no clipping. Public sample count: 961.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-16T19:00:13Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

Typed vision decisions on CharXiv; private population.

Vision board identity clef-omni. Effective image budget 1.6 MP.

Vision board 0.3.2, 50% public and 50% private; public/private are distinct from the text board. Source benchmark weights and image caps apply. author cap 1,048,576 px lifted to the board cap via the documented media_kwargs (processor max 12.8 MP) Private normalized scores may be below zero; no clipping. Public sample count: 532.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-16T19:00:13Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

Typed vision decisions on CharXiv; pub population.

Vision board identity clef-omni. Effective image budget 1.6 MP.

Vision board 0.3.2, 50% public and 50% private; public/private are distinct from the text board. Source benchmark weights and image caps apply. author cap 1,048,576 px lifted to the board cap via the documented media_kwargs (processor max 12.8 MP) Private normalized scores may be below zero; no clipping. Public sample count: 532.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-16T19:00:13Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

Typed vision decisions on CV-Bench; private population.

Vision board identity clef-omni. Effective image budget 1.6 MP.

Vision board 0.3.2, 50% public and 50% private; public/private are distinct from the text board. Source benchmark weights and image caps apply. author cap 1,048,576 px lifted to the board cap via the documented media_kwargs (processor max 12.8 MP) Private normalized scores may be below zero; no clipping. Public sample count: 2038.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-16T19:00:13Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

Typed vision decisions on CV-Bench; pub population.

Vision board identity clef-omni. Effective image budget 1.6 MP.

Vision board 0.3.2, 50% public and 50% private; public/private are distinct from the text board. Source benchmark weights and image caps apply. author cap 1,048,576 px lifted to the board cap via the documented media_kwargs (processor max 12.8 MP) Private normalized scores may be below zero; no clipping. Public sample count: 2038.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-16T19:00:13Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

Multimodal typed-choice vision evaluation · Full

Vision board identity clef-omni. Effective image budget 1.6 MP.

Vision board 0.3.2, 50% public and 50% private; public/private are distinct from the text board. Source benchmark weights and image caps apply. author cap 1,048,576 px lifted to the board cap via the documented media_kwargs (processor max 12.8 MP)

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-16T19:00:13Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

Typed vision decisions on InfographicVQA; private population.

Vision board identity clef-omni. Effective image budget 1.6 MP.

Vision board 0.3.2, 50% public and 50% private; public/private are distinct from the text board. Source benchmark weights and image caps apply. author cap 1,048,576 px lifted to the board cap via the documented media_kwargs (processor max 12.8 MP) Private normalized scores may be below zero; no clipping. Public sample count: 244.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-16T19:00:13Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

Typed vision decisions on InfographicVQA; pub population.

Vision board identity clef-omni. Effective image budget 1.6 MP.

Vision board 0.3.2, 50% public and 50% private; public/private are distinct from the text board. Source benchmark weights and image caps apply. author cap 1,048,576 px lifted to the board cap via the documented media_kwargs (processor max 12.8 MP) Private normalized scores may be below zero; no clipping. Public sample count: 244.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-16T19:00:13Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

Typed vision decisions on KIE (CORD+FUNSD); private population.

Vision board identity clef-omni. Effective image budget 1.6 MP.

Vision board 0.3.2, 50% public and 50% private; public/private are distinct from the text board. Source benchmark weights and image caps apply. author cap 1,048,576 px lifted to the board cap via the documented media_kwargs (processor max 12.8 MP) Private normalized scores may be below zero; no clipping. Public sample count: 2465.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-16T19:00:13Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

Typed vision decisions on KIE (CORD+FUNSD); pub population.

Vision board identity clef-omni. Effective image budget 1.6 MP.

Vision board 0.3.2, 50% public and 50% private; public/private are distinct from the text board. Source benchmark weights and image caps apply. author cap 1,048,576 px lifted to the board cap via the documented media_kwargs (processor max 12.8 MP) Private normalized scores may be below zero; no clipping. Public sample count: 2465.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-16T19:00:13Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

Typed vision decisions on Mind2Web; private population.

Vision board identity clef-omni. Effective image budget 1.6 MP.

Vision board 0.3.2, 50% public and 50% private; public/private are distinct from the text board. Source benchmark weights and image caps apply. author cap 1,048,576 px lifted to the board cap via the documented media_kwargs (processor max 12.8 MP) Private normalized scores may be below zero; no clipping. Public sample count: 973.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-16T19:00:13Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

Typed vision decisions on Mind2Web; pub population.

Vision board identity clef-omni. Effective image budget 1.6 MP.

Vision board 0.3.2, 50% public and 50% private; public/private are distinct from the text board. Source benchmark weights and image caps apply. author cap 1,048,576 px lifted to the board cap via the documented media_kwargs (processor max 12.8 MP) Private normalized scores may be below zero; no clipping. Public sample count: 973.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-16T19:00:13Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

Typed vision decisions on MMMU-Pro vision; pub population.

Vision board identity clef-omni. Effective image budget 1.6 MP.

Vision board 0.3.2, 50% public and 50% private; public/private are distinct from the text board. Source benchmark weights and image caps apply. author cap 1,048,576 px lifted to the board cap via the documented media_kwargs (processor max 12.8 MP) Private normalized scores may be below zero; no clipping. Public sample count: 1730.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-16T19:00:13Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

Typed vision decisions on Moderation (Hateful Memes); private population.

Vision board identity clef-omni. Effective image budget 1.6 MP.

Vision board 0.3.2, 50% public and 50% private; public/private are distinct from the text board. Source benchmark weights and image caps apply. author cap 1,048,576 px lifted to the board cap via the documented media_kwargs (processor max 12.8 MP) Private normalized scores may be below zero; no clipping. Public sample count: 1213.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-16T19:00:13Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

Typed vision decisions on Moderation (Hateful Memes); pub population.

Vision board identity clef-omni. Effective image budget 1.6 MP.

Vision board 0.3.2, 50% public and 50% private; public/private are distinct from the text board. Source benchmark weights and image caps apply. author cap 1,048,576 px lifted to the board cap via the documented media_kwargs (processor max 12.8 MP) Private normalized scores may be below zero; no clipping. Public sample count: 1213.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-16T19:00:13Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

Multimodal typed-choice vision evaluation · Private

Vision board identity clef-omni. Effective image budget 1.6 MP.

Vision board 0.3.2, 50% public and 50% private; public/private are distinct from the text board. Source benchmark weights and image caps apply. author cap 1,048,576 px lifted to the board cap via the documented media_kwargs (processor max 12.8 MP)

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-16T19:00:13Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

Multimodal typed-choice vision evaluation · Public

Vision board identity clef-omni. Effective image budget 1.6 MP.

Vision board 0.3.2, 50% public and 50% private; public/private are distinct from the text board. Source benchmark weights and image caps apply. author cap 1,048,576 px lifted to the board cap via the documented media_kwargs (processor max 12.8 MP)

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-16T19:00:13Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

Typed vision decisions on R-Bench-M; pub population.

Vision board identity clef-omni. Effective image budget 1.6 MP.

Vision board 0.3.2, 50% public and 50% private; public/private are distinct from the text board. Source benchmark weights and image caps apply. author cap 1,048,576 px lifted to the board cap via the documented media_kwargs (processor max 12.8 MP) Private normalized scores may be below zero; no clipping. Public sample count: 665.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-16T19:00:13Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

Typed vision decisions on RealWorldQA; private population.

Vision board identity clef-omni. Effective image budget 1.6 MP.

Vision board 0.3.2, 50% public and 50% private; public/private are distinct from the text board. Source benchmark weights and image caps apply. author cap 1,048,576 px lifted to the board cap via the documented media_kwargs (processor max 12.8 MP) Private normalized scores may be below zero; no clipping. Public sample count: 589.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-16T19:00:13Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

Typed vision decisions on RealWorldQA; pub population.

Vision board identity clef-omni. Effective image budget 1.6 MP.

Vision board 0.3.2, 50% public and 50% private; public/private are distinct from the text board. Source benchmark weights and image caps apply. author cap 1,048,576 px lifted to the board cap via the documented media_kwargs (processor max 12.8 MP) Private normalized scores may be below zero; no clipping. Public sample count: 589.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-16T19:00:13Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

Typed vision decisions on Winoground; private population.

Vision board identity clef-omni. Effective image budget 1.6 MP.

Vision board 0.3.2, 50% public and 50% private; public/private are distinct from the text board. Source benchmark weights and image caps apply. author cap 1,048,576 px lifted to the board cap via the documented media_kwargs (processor max 12.8 MP) Private normalized scores may be below zero; no clipping. Public sample count: 800.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-16T19:00:13Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

Typed vision decisions on Winoground; pub population.

Vision board identity clef-omni. Effective image budget 1.6 MP.

Vision board 0.3.2, 50% public and 50% private; public/private are distinct from the text board. Source benchmark weights and image caps apply. author cap 1,048,576 px lifted to the board cap via the documented media_kwargs (processor max 12.8 MP) Private normalized scores may be below zero; no clipping. Public sample count: 800.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-16T19:00:13Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

JevBench Binary slices: aegis2 · acc @0.5

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 84.8%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: aegis2 · acc if flipped

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 15.2%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: aegis2 · AUROC

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.9082.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: aegis2 · Brier

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.1232.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: aegis2 · ECE

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.0755.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: aegis2 · FPR

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 16.8%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: aegis2 · positive recall

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 86.4%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: aegis2_response · acc @0.5

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 79.0%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: aegis2_response · acc if flipped

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 21.0%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: aegis2_response · AUROC

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.8850.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: aegis2_response · Brier

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.1496.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: aegis2_response · ECE

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.0808.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: aegis2_response · FPR

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 18.0%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: aegis2_response · positive recall

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 76.0%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: atbench500 · acc @0.5

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 67.4%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: atbench500 · acc if flipped

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 32.6%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: atbench500 · AUROC

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.8990.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: atbench500 · Brier

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.2253.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: atbench500 · ECE

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.1636.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: atbench500 · FPR

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 1.2%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: atbench500 · positive recall

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 36.0%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: jailbreak_classification · acc @0.5

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 92.5%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 400

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: jailbreak_classification · acc if flipped

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 7.5%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 400

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: jailbreak_classification · AUROC

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.9803.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 400

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: jailbreak_classification · Brier

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.0517.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 400

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: jailbreak_classification · ECE

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.0358.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 400

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: jailbreak_classification · FPR

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 6.9%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 400

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: jailbreak_classification · positive recall

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 91.5%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 400

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: prompt_injections · acc @0.5

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 71.6%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 116

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: prompt_injections · acc if flipped

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 28.4%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 116

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: prompt_injections · AUROC

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.9872.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 116

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: prompt_injections · Brier

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.1829.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 116

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: prompt_injections · ECE

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.1564.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 116

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: prompt_injections · FPR

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.0%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 116

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: prompt_injections · positive recall

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 45.0%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 116

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench By subset: rewardbench2 · accuracy

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 47.5%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 240

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench By subset: rewardbench2 · accuracy

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 64.2%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 240

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench By subset: rewardbench2 · accuracy

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 61.7%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 240

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench By subset: rewardbench2 · accuracy

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 25.0%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 240

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench By subset: rewardbench2 · accuracy

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 81.7%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 240

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: banking77 · accuracy

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 94.8%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 1000

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: banking77 · Brier

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.1272.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 1000

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: banking77 · ECE (10 bins)

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.1887.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 1000

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: banking77 · macro-F1

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.9458.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 1000

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: banking77 · NLL

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.4028.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 1000

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: mcpbench_server_routing · accuracy

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 85.7%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 112

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: mcpbench_server_routing · Brier

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.2392.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 112

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: mcpbench_server_routing · ECE (10 bins)

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.2088.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 112

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: mcpbench_server_routing · macro-F1

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.8347.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 112

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: mcpbench_server_routing · NLL

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.6420.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 112

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: mcpbench_tool_selection · accuracy

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 92.3%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 26

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: mcpbench_tool_selection · Brier

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.1568.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 26

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: mcpbench_tool_selection · ECE (10 bins)

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.2766.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 26

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: mcpbench_tool_selection · macro-F1

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.8974.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 26

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: mcpbench_tool_selection · NLL

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.3941.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 26

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: mcpmark_server_routing · accuracy

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 46.7%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 338

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: mcpmark_server_routing · Brier

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.6945.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 338

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: mcpmark_server_routing · ECE (10 bins)

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.1896.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 338

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: mcpmark_server_routing · macro-F1

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.4883.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 338

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: mcpmark_server_routing · NLL

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 1.4538.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 338

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: mcpmark_tool_selection · accuracy

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 100.0%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 18

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: mcpmark_tool_selection · Brier

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.0167.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 18

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: mcpmark_tool_selection · ECE (10 bins)

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.1024.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 18

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: mcpmark_tool_selection · macro-F1

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 1.0000.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 18

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: mcpmark_tool_selection · NLL

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.1093.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 18

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: medmcqa · accuracy

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 72.5%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 1000

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: medmcqa · Brier

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.3757.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 1000

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: medmcqa · ECE (10 bins)

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.0332.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 1000

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: medmcqa · macro-F1

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.7252.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 1000

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: medmcqa · NLL

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.7253.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 1000

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: medqa_usmle · accuracy

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 79.5%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 1000

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: medqa_usmle · Brier

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.2940.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 1000

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: medqa_usmle · ECE (10 bins)

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.0479.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 1000

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: medqa_usmle · macro-F1

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.7951.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 1000

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: medqa_usmle · NLL

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.5774.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 1000

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: mmlu_pro · accuracy

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 71.5%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 1000

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: mmlu_pro · Brier

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.3944.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 1000

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: mmlu_pro · ECE (10 bins)

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.0712.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 1000

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: mmlu_pro · macro-F1

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.7117.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 1000

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: mmlu_pro · NLL

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.9462.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 1000

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: pubmedqa · accuracy

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 67.4%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: pubmedqa · Brier

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.4888.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: pubmedqa · ECE (10 bins)

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.1761.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: pubmedqa · macro-F1

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.5542.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: pubmedqa · NLL

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.8908.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: rewardbench2 · accuracy

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 56.0%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 1200

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: rewardbench2 · Brier

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.5909.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 1200

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: rewardbench2 · ECE (10 bins)

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.0701.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 1200

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: rewardbench2 · macro-F1

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.5605.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 1200

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: rewardbench2 · NLL

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 1.1273.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 1200

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: scienceqa_text · accuracy

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 92.1%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 1000

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: scienceqa_text · Brier

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.1091.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 1000

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: scienceqa_text · ECE (10 bins)

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.0465.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 1000

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: scienceqa_text · macro-F1

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.9291.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 1000

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: scienceqa_text · NLL

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.1974.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 1000

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Headline: aegis2 · accuracy

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.8480.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 500 · Source 95% CI: 81.6–88

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Headline: aegis2_response · accuracy

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.7900.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 500 · Source 95% CI: 75.4–82.6

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Headline: atbench500 · accuracy

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.6740.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 500 · Source 95% CI: 63.4–71.8

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Headline: banking77 · accuracy

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.9480.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 1000 · Source 95% CI: 92.80000000000001–96.6

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Headline: helpsteer2_helpfulness · mae_expected_level

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 1.1024.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 500 · Source 95% CI: 1.026–1.182

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Headline: jailbreak_classification · accuracy

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.9250.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 400 · Source 95% CI: 89.7–95

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Headline: mcpbench_server_routing · server_routing_accuracy

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.8571.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 112 · Source 95% CI: 75–94.6

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Headline: mcpbench_tool_selection · tool_selection_accuracy

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.9231.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 26

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Headline: mcpmark_server_routing · server_routing_accuracy

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.4675.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 338 · Source 95% CI: 39.1–53.800000000000004

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Headline: mcpmark_tool_selection · tool_selection_accuracy

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 1.0000.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 18

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Headline: medmcqa · accuracy

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.7250.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 1000 · Source 95% CI: 68.8–76.3

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Headline: medqa_usmle · accuracy

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.7950.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 1000 · Source 95% CI: 76.1–82.8

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Headline: mmlu_pro · accuracy

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.7150.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 1000 · Source 95% CI: 67.60000000000001–75.1

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Headline: nfcorpus_rank_choice · ndcg@10

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.3573.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 298 · Source 95% CI: 32.4–39.300000000000004

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Headline: nfcorpus_rank_graded · ndcg@10

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.3797.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 4924 · Source 95% CI: 34.599999999999994–41.6

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Headline: nfcorpus_rank_noul · ndcg@10

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.3612.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 298 · Source 95% CI: 32.5–39.6

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Headline: prompt_injections · accuracy

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.7155.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 116 · Source 95% CI: 62.9–79.3

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Headline: pubmedqa · accuracy

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.6740.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 500 · Source 95% CI: 63.2–71.6

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Headline: rewardbench2 · accuracy

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.5600.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 1200 · Source 95% CI: 52.2–59.4

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Headline: scienceqa_text · accuracy

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.9210.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 1000 · Source 95% CI: 89.9–94.19999999999999

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Headline: scifact_rank_choice · ndcg@10

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.7233.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 300 · Source 95% CI: 68.2–76.8

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Headline: scifact_rank_noul · ndcg@10

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.7162.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 300 · Source 95% CI: 66.9–76

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Headline: sst5 · mae_expected_level

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.5233.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 500 · Source 95% CI: 0.487–0.559

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: banking77 · both correct

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 94.8%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: banking77 · flipped by reordering

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.0%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: banking77 · original order

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 94.8%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: banking77 · permuted order

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 94.8%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: mcpbench_server_routing · both correct

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 85.7%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 56

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: mcpbench_server_routing · flipped by reordering

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.0%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 56

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: mcpbench_server_routing · original order

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 85.7%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 56

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: mcpbench_server_routing · permuted order

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 85.7%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 56

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: mcpbench_tool_selection · both correct

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 100.0%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 4

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: mcpbench_tool_selection · flipped by reordering

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.0%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 4

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: mcpbench_tool_selection · original order

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 92.3%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 4

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: mcpbench_tool_selection · permuted order

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 92.3%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 4

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: mcpmark_server_routing · both correct

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 46.7%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 169

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: mcpmark_server_routing · flipped by reordering

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.0%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 169

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: mcpmark_server_routing · original order

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 46.7%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 169

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: mcpmark_server_routing · permuted order

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 46.7%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 169

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: mcpmark_tool_selection · both correct

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 100.0%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 3

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: mcpmark_tool_selection · flipped by reordering

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.0%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 3

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: mcpmark_tool_selection · original order

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 100.0%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 3

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: mcpmark_tool_selection · permuted order

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 100.0%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 3

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: medmcqa · both correct

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 69.0%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: medmcqa · flipped by reordering

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 7.0%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: medmcqa · original order

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 72.2%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: medmcqa · permuted order

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 72.8%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: medqa_usmle · both correct

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 76.4%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: medqa_usmle · flipped by reordering

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 6.2%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: medqa_usmle · original order

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 79.2%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: medqa_usmle · permuted order

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 79.8%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: mmlu_pro · both correct

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 65.6%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: mmlu_pro · flipped by reordering

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 11.8%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: mmlu_pro · original order

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 71.4%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: mmlu_pro · permuted order

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 71.6%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: rewardbench2 · both correct

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 48.8%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 594

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: rewardbench2 · flipped by reordering

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 14.5%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 594

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: rewardbench2 · original order

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 55.2%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 594

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: rewardbench2 · permuted order

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 56.8%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 594

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: scienceqa_text · both correct

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 89.4%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: scienceqa_text · flipped by reordering

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 5.4%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: scienceqa_text · original order

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 92.4%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: scienceqa_text · permuted order

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 91.8%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ordinal slices: helpsteer2_helpfulness · exact (rounded)

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 25.8%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ordinal slices: helpsteer2_helpfulness · MAE (expected level)

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 1.102.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ordinal slices: helpsteer2_helpfulness · RPS

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.2048.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ordinal slices: helpsteer2_helpfulness · Spearman

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.444.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ordinal slices: helpsteer2_helpfulness · within 1 level

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 72.2%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ordinal slices: nfcorpus_rank_graded · exact (rounded)

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 64.6%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 4924

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ordinal slices: nfcorpus_rank_graded · MAE (expected level)

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.443.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 4924

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ordinal slices: nfcorpus_rank_graded · RPS

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.1470.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 4924

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ordinal slices: nfcorpus_rank_graded · Spearman

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.490.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 4924

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ordinal slices: nfcorpus_rank_graded · within 1 level

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 94.6%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 4924

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ordinal slices: sst5 · exact (rounded)

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 57.6%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ordinal slices: sst5 · MAE (expected level)

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.523.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ordinal slices: sst5 · RPS

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.0827.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ordinal slices: sst5 · Spearman

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.880.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ordinal slices: sst5 · within 1 level

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 97.2%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ranking slices: nfcorpus_rank_choice · MRR@10

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.5967.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 288

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ranking slices: nfcorpus_rank_choice · nDCG@10

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.3573.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 288

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ranking slices: nfcorpus_rank_choice · Recall@10

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.1632.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 288

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ranking slices: nfcorpus_rank_graded · MRR@10

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.6291.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 298

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ranking slices: nfcorpus_rank_graded · nDCG@10

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.3797.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 298

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ranking slices: nfcorpus_rank_graded · Recall@10

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.1747.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 298

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ranking slices: nfcorpus_rank_noul · MRR@10

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.5983.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 298

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ranking slices: nfcorpus_rank_noul · nDCG@10

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.3612.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 298

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ranking slices: nfcorpus_rank_noul · Recall@10

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.1713.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 298

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ranking slices: scifact_rank_choice · MRR@10

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.7075.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 300

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ranking slices: scifact_rank_choice · nDCG@10

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.7233.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 300

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ranking slices: scifact_rank_choice · Recall@10

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.8017.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 300

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ranking slices: scifact_rank_noul · MRR@10

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.6946.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 300

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ranking slices: scifact_rank_noul · nDCG@10

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.7162.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 300

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ranking slices: scifact_rank_noul · Recall@10

Scored 2026-10-15T23:38:40-0700. Model requested `or-cloudflare-clef-omni`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.8159.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:40-0700 · Samples: 300

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

Decision Index · ACOS (per-review F1) as reported by Cloudflare

Card Cloudflare/clef-omni at 0db1cd2607d76a7bdb2a382f659e7b313079f84b; table Decision Index.

Published table label: ACOS (per-review F1).

Vendor-published observation, distinct from independent and other vendor harnesses. Cloudflare internal Decision Index 0.2.1 and workflow cohorts; ForecastBench values are native published Brier ×100, not accuracy.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-01 · Revision: 0b331204bb13fbd2ca93a64df1956f5b55478ce5 · Published observations; result reuse terms not specified

Decision Index · Amazon ESCI (macro-F1) as reported by Cloudflare

Card Cloudflare/clef-omni at 0db1cd2607d76a7bdb2a382f659e7b313079f84b; table Decision Index.

Published table label: Amazon ESCI (macro-F1).

Vendor-published observation, distinct from independent and other vendor harnesses. Cloudflare internal Decision Index 0.2.1 and workflow cohorts; ForecastBench values are native published Brier ×100, not accuracy.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-01 · Revision: 0b331204bb13fbd2ca93a64df1956f5b55478ce5 · Published observations; result reuse terms not specified

Decision Index · ANLI (macro-F1) as reported by Cloudflare

Card Cloudflare/clef-omni at 0db1cd2607d76a7bdb2a382f659e7b313079f84b; table Decision Index.

Published table label: ANLI (macro-F1).

Vendor-published observation, distinct from independent and other vendor harnesses. Cloudflare internal Decision Index 0.2.1 and workflow cohorts; ForecastBench values are native published Brier ×100, not accuracy.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-01 · Revision: 0b331204bb13fbd2ca93a64df1956f5b55478ce5 · Published observations; result reuse terms not specified

Decision Index · API-Bank (accuracy) as reported by Cloudflare

Card Cloudflare/clef-omni at 0db1cd2607d76a7bdb2a382f659e7b313079f84b; table Decision Index.

Published table label: API-Bank (accuracy).

Vendor-published observation, distinct from independent and other vendor harnesses. Cloudflare internal Decision Index 0.2.1 and workflow cohorts; ForecastBench values are native published Brier ×100, not accuracy.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-01 · Revision: 0b331204bb13fbd2ca93a64df1956f5b55478ce5 · Published observations; result reuse terms not specified

Decision Index · ARC-Challenge (accuracy) as reported by Cloudflare

Card Cloudflare/clef-omni at 0db1cd2607d76a7bdb2a382f659e7b313079f84b; table Decision Index.

Published table label: ARC-Challenge (accuracy).

Vendor-published observation, distinct from independent and other vendor harnesses. Cloudflare internal Decision Index 0.2.1 and workflow cohorts; ForecastBench values are native published Brier ×100, not accuracy.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-01 · Revision: 0b331204bb13fbd2ca93a64df1956f5b55478ce5 · Published observations; result reuse terms not specified

Decision Index · ARC-Easy (accuracy) as reported by Cloudflare

Card Cloudflare/clef-omni at 0db1cd2607d76a7bdb2a382f659e7b313079f84b; table Decision Index.

Published table label: ARC-Easy (accuracy).

Vendor-published observation, distinct from independent and other vendor harnesses. Cloudflare internal Decision Index 0.2.1 and workflow cohorts; ForecastBench values are native published Brier ×100, not accuracy.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-01 · Revision: 0b331204bb13fbd2ca93a64df1956f5b55478ce5 · Published observations; result reuse terms not specified

Decision Index · BANKING77 (macro-F1) as reported by Cloudflare

Card Cloudflare/clef-omni at 0db1cd2607d76a7bdb2a382f659e7b313079f84b; table Decision Index.

Published table label: BANKING77 (macro-F1).

Vendor-published observation, distinct from independent and other vendor harnesses. Cloudflare internal Decision Index 0.2.1 and workflow cohorts; ForecastBench values are native published Brier ×100, not accuracy.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-01 · Revision: 0b331204bb13fbd2ca93a64df1956f5b55478ce5 · Published observations; result reuse terms not specified

Decision Index · BBH (accuracy) as reported by Cloudflare

Card Cloudflare/clef-omni at 0db1cd2607d76a7bdb2a382f659e7b313079f84b; table Decision Index.

Published table label: BBH (accuracy).

Vendor-published observation, distinct from independent and other vendor harnesses. Cloudflare internal Decision Index 0.2.1 and workflow cohorts; ForecastBench values are native published Brier ×100, not accuracy.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-01 · Revision: 0b331204bb13fbd2ca93a64df1956f5b55478ce5 · Published observations; result reuse terms not specified

Decision Index · BFCL (case exact accuracy) as reported by Cloudflare

Card Cloudflare/clef-omni at 0db1cd2607d76a7bdb2a382f659e7b313079f84b; table Decision Index.

Published table label: BFCL (case exact accuracy).

Vendor-published observation, distinct from independent and other vendor harnesses. Cloudflare internal Decision Index 0.2.1 and workflow cohorts; ForecastBench values are native published Brier ×100, not accuracy.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-01 · Revision: 0b331204bb13fbd2ca93a64df1956f5b55478ce5 · Published observations; result reuse terms not specified

Decision Index · BPoMP (accuracy) as reported by Cloudflare

Card Cloudflare/clef-omni at 0db1cd2607d76a7bdb2a382f659e7b313079f84b; table Decision Index.

Published table label: BPoMP (accuracy).

Vendor-published observation, distinct from independent and other vendor harnesses. Cloudflare internal Decision Index 0.2.1 and workflow cohorts; ForecastBench values are native published Brier ×100, not accuracy.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-01 · Revision: 0b331204bb13fbd2ca93a64df1956f5b55478ce5 · Published observations; result reuse terms not specified

Decision Index · BRIGHT (nDCG@10) as reported by Cloudflare

Card Cloudflare/clef-omni at 0db1cd2607d76a7bdb2a382f659e7b313079f84b; table Decision Index.

Published table label: BRIGHT (nDCG@10).

Vendor-published observation, distinct from independent and other vendor harnesses. Cloudflare internal Decision Index 0.2.1 and workflow cohorts; ForecastBench values are native published Brier ×100, not accuracy.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-01 · Revision: 0b331204bb13fbd2ca93a64df1956f5b55478ce5 · Published observations; result reuse terms not specified

Decision Index · cfcolor (accuracy) as reported by Cloudflare

Card Cloudflare/clef-omni at 0db1cd2607d76a7bdb2a382f659e7b313079f84b; table Decision Index.

Published table label: cfcolor (accuracy).

Vendor-published observation, distinct from independent and other vendor harnesses. Cloudflare internal Decision Index 0.2.1 and workflow cohorts; ForecastBench values are native published Brier ×100, not accuracy.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-01 · Revision: 0b331204bb13fbd2ca93a64df1956f5b55478ce5 · Published observations; result reuse terms not specified

Decision Index · ChessBench (accuracy) as reported by Cloudflare

Card Cloudflare/clef-omni at 0db1cd2607d76a7bdb2a382f659e7b313079f84b; table Decision Index.

Published table label: ChessBench (accuracy).

Vendor-published observation, distinct from independent and other vendor harnesses. Cloudflare internal Decision Index 0.2.1 and workflow cohorts; ForecastBench values are native published Brier ×100, not accuracy.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-01 · Revision: 0b331204bb13fbd2ca93a64df1956f5b55478ce5 · Published observations; result reuse terms not specified

Decision Index · CLadder (accuracy) as reported by Cloudflare

Card Cloudflare/clef-omni at 0db1cd2607d76a7bdb2a382f659e7b313079f84b; table Decision Index.

Published table label: CLadder (accuracy).

Vendor-published observation, distinct from independent and other vendor harnesses. Cloudflare internal Decision Index 0.2.1 and workflow cohorts; ForecastBench values are native published Brier ×100, not accuracy.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-01 · Revision: 0b331204bb13fbd2ca93a64df1956f5b55478ce5 · Published observations; result reuse terms not specified

Decision Index · CLINC150+OOS (macro-F1) as reported by Cloudflare

Card Cloudflare/clef-omni at 0db1cd2607d76a7bdb2a382f659e7b313079f84b; table Decision Index.

Published table label: CLINC150+OOS (macro-F1).

Vendor-published observation, distinct from independent and other vendor harnesses. Cloudflare internal Decision Index 0.2.1 and workflow cohorts; ForecastBench values are native published Brier ×100, not accuracy.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-01 · Revision: 0b331204bb13fbd2ca93a64df1956f5b55478ce5 · Published observations; result reuse terms not specified

Decision Index · ContractNLI (macro-F1) as reported by Cloudflare

Card Cloudflare/clef-omni at 0db1cd2607d76a7bdb2a382f659e7b313079f84b; table Decision Index.

Published table label: ContractNLI (macro-F1).

Vendor-published observation, distinct from independent and other vendor harnesses. Cloudflare internal Decision Index 0.2.1 and workflow cohorts; ForecastBench values are native published Brier ×100, not accuracy.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-01 · Revision: 0b331204bb13fbd2ca93a64df1956f5b55478ce5 · Published observations; result reuse terms not specified

Decision Index · CRUXEval (accuracy) as reported by Cloudflare

Card Cloudflare/clef-omni at 0db1cd2607d76a7bdb2a382f659e7b313079f84b; table Decision Index.

Published table label: CRUXEval (accuracy).

Vendor-published observation, distinct from independent and other vendor harnesses. Cloudflare internal Decision Index 0.2.1 and workflow cohorts; ForecastBench values are native published Brier ×100, not accuracy.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-01 · Revision: 0b331204bb13fbd2ca93a64df1956f5b55478ce5 · Published observations; result reuse terms not specified

Decision Index · FinEntity (macro-F1) as reported by Cloudflare

Card Cloudflare/clef-omni at 0db1cd2607d76a7bdb2a382f659e7b313079f84b; table Decision Index.

Published table label: FinEntity (macro-F1).

Vendor-published observation, distinct from independent and other vendor harnesses. Cloudflare internal Decision Index 0.2.1 and workflow cohorts; ForecastBench values are native published Brier ×100, not accuracy.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-01 · Revision: 0b331204bb13fbd2ca93a64df1956f5b55478ce5 · Published observations; result reuse terms not specified

Decision Index · ForecastBench (Brier, lower is better) as reported by Cloudflare

Card Cloudflare/clef-omni at 0db1cd2607d76a7bdb2a382f659e7b313079f84b; table Decision Index.

Published table label: ForecastBench (Brier, lower is better).

Vendor-published observation, distinct from independent and other vendor harnesses. Cloudflare internal Decision Index 0.2.1 and workflow cohorts; ForecastBench values are native published Brier ×100, not accuracy.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-01 · Revision: 0b331204bb13fbd2ca93a64df1956f5b55478ce5 · Published observations; result reuse terms not specified

Decision Index · GPQA Diamond (accuracy) as reported by Cloudflare

Card Cloudflare/clef-omni at 0db1cd2607d76a7bdb2a382f659e7b313079f84b; table Decision Index.

Published table label: GPQA Diamond (accuracy).

Vendor-published observation, distinct from independent and other vendor harnesses. Cloudflare internal Decision Index 0.2.1 and workflow cohorts; ForecastBench values are native published Brier ×100, not accuracy.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-01 · Revision: 0b331204bb13fbd2ca93a64df1956f5b55478ce5 · Published observations; result reuse terms not specified

Decision Index · GSM8K (accuracy) as reported by Cloudflare

Card Cloudflare/clef-omni at 0db1cd2607d76a7bdb2a382f659e7b313079f84b; table Decision Index.

Published table label: GSM8K (accuracy).

Vendor-published observation, distinct from independent and other vendor harnesses. Cloudflare internal Decision Index 0.2.1 and workflow cohorts; ForecastBench values are native published Brier ×100, not accuracy.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-01 · Revision: 0b331204bb13fbd2ca93a64df1956f5b55478ce5 · Published observations; result reuse terms not specified

Decision Index · Habermas Machine (accuracy) as reported by Cloudflare

Card Cloudflare/clef-omni at 0db1cd2607d76a7bdb2a382f659e7b313079f84b; table Decision Index.

Published table label: Habermas Machine (accuracy).

Vendor-published observation, distinct from independent and other vendor harnesses. Cloudflare internal Decision Index 0.2.1 and workflow cohorts; ForecastBench values are native published Brier ×100, not accuracy.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-01 · Revision: 0b331204bb13fbd2ca93a64df1956f5b55478ce5 · Published observations; result reuse terms not specified

Decision Index · HellaSwag (accuracy) as reported by Cloudflare

Card Cloudflare/clef-omni at 0db1cd2607d76a7bdb2a382f659e7b313079f84b; table Decision Index.

Published table label: HellaSwag (accuracy).

Vendor-published observation, distinct from independent and other vendor harnesses. Cloudflare internal Decision Index 0.2.1 and workflow cohorts; ForecastBench values are native published Brier ×100, not accuracy.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-01 · Revision: 0b331204bb13fbd2ca93a64df1956f5b55478ce5 · Published observations; result reuse terms not specified

Decision Index · Home appliance simulator (case exact accuracy) as reported by Cloudflare

Card Cloudflare/clef-omni at 0db1cd2607d76a7bdb2a382f659e7b313079f84b; table Decision Index.

Published table label: Home appliance simulator (case exact accuracy).

Vendor-published observation, distinct from independent and other vendor harnesses. Cloudflare internal Decision Index 0.2.1 and workflow cohorts; ForecastBench values are native published Brier ×100, not accuracy.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-01 · Revision: 0b331204bb13fbd2ca93a64df1956f5b55478ce5 · Published observations; result reuse terms not specified

Decision Index · HoVer (accuracy) as reported by Cloudflare

Card Cloudflare/clef-omni at 0db1cd2607d76a7bdb2a382f659e7b313079f84b; table Decision Index.

Published table label: HoVer (accuracy).

Vendor-published observation, distinct from independent and other vendor harnesses. Cloudflare internal Decision Index 0.2.1 and workflow cohorts; ForecastBench values are native published Brier ×100, not accuracy.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-01 · Revision: 0b331204bb13fbd2ca93a64df1956f5b55478ce5 · Published observations; result reuse terms not specified

Decision Index · Humicroedit (accuracy) as reported by Cloudflare

Card Cloudflare/clef-omni at 0db1cd2607d76a7bdb2a382f659e7b313079f84b; table Decision Index.

Published table label: Humicroedit (accuracy).

Vendor-published observation, distinct from independent and other vendor harnesses. Cloudflare internal Decision Index 0.2.1 and workflow cohorts; ForecastBench values are native published Brier ×100, not accuracy.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-01 · Revision: 0b331204bb13fbd2ca93a64df1956f5b55478ce5 · Published observations; result reuse terms not specified

Decision Index · MMLU (accuracy) as reported by Cloudflare

Card Cloudflare/clef-omni at 0db1cd2607d76a7bdb2a382f659e7b313079f84b; table Decision Index.

Published table label: MMLU (accuracy).

Vendor-published observation, distinct from independent and other vendor harnesses. Cloudflare internal Decision Index 0.2.1 and workflow cohorts; ForecastBench values are native published Brier ×100, not accuracy.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-01 · Revision: 0b331204bb13fbd2ca93a64df1956f5b55478ce5 · Published observations; result reuse terms not specified

Decision Index · MMLU-Pro (accuracy) as reported by Cloudflare

Card Cloudflare/clef-omni at 0db1cd2607d76a7bdb2a382f659e7b313079f84b; table Decision Index.

Published table label: MMLU-Pro (accuracy).

Vendor-published observation, distinct from independent and other vendor harnesses. Cloudflare internal Decision Index 0.2.1 and workflow cohorts; ForecastBench values are native published Brier ×100, not accuracy.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-01 · Revision: 0b331204bb13fbd2ca93a64df1956f5b55478ce5 · Published observations; result reuse terms not specified

Decision Index · MuSR (accuracy) as reported by Cloudflare

Card Cloudflare/clef-omni at 0db1cd2607d76a7bdb2a382f659e7b313079f84b; table Decision Index.

Published table label: MuSR (accuracy).

Vendor-published observation, distinct from independent and other vendor harnesses. Cloudflare internal Decision Index 0.2.1 and workflow cohorts; ForecastBench values are native published Brier ×100, not accuracy.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-01 · Revision: 0b331204bb13fbd2ca93a64df1956f5b55478ce5 · Published observations; result reuse terms not specified

Decision Index · New Yorker (accuracy) as reported by Cloudflare

Card Cloudflare/clef-omni at 0db1cd2607d76a7bdb2a382f659e7b313079f84b; table Decision Index.

Published table label: New Yorker (accuracy).

Vendor-published observation, distinct from independent and other vendor harnesses. Cloudflare internal Decision Index 0.2.1 and workflow cohorts; ForecastBench values are native published Brier ×100, not accuracy.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-01 · Revision: 0b331204bb13fbd2ca93a64df1956f5b55478ce5 · Published observations; result reuse terms not specified

Decision Index · NLI4CT (macro-F1) as reported by Cloudflare

Card Cloudflare/clef-omni at 0db1cd2607d76a7bdb2a382f659e7b313079f84b; table Decision Index.

Published table label: NLI4CT (macro-F1).

Vendor-published observation, distinct from independent and other vendor harnesses. Cloudflare internal Decision Index 0.2.1 and workflow cohorts; ForecastBench values are native published Brier ×100, not accuracy.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-01 · Revision: 0b331204bb13fbd2ca93a64df1956f5b55478ce5 · Published observations; result reuse terms not specified

Decision Index · PhishNChips (accuracy) as reported by Cloudflare

Card Cloudflare/clef-omni at 0db1cd2607d76a7bdb2a382f659e7b313079f84b; table Decision Index.

Published table label: PhishNChips (accuracy).

Vendor-published observation, distinct from independent and other vendor harnesses. Cloudflare internal Decision Index 0.2.1 and workflow cohorts; ForecastBench values are native published Brier ×100, not accuracy.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-01 · Revision: 0b331204bb13fbd2ca93a64df1956f5b55478ce5 · Published observations; result reuse terms not specified

Decision Index · POP909-CL (accuracy) as reported by Cloudflare

Card Cloudflare/clef-omni at 0db1cd2607d76a7bdb2a382f659e7b313079f84b; table Decision Index.

Published table label: POP909-CL (accuracy).

Vendor-published observation, distinct from independent and other vendor harnesses. Cloudflare internal Decision Index 0.2.1 and workflow cohorts; ForecastBench values are native published Brier ×100, not accuracy.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-01 · Revision: 0b331204bb13fbd2ca93a64df1956f5b55478ce5 · Published observations; result reuse terms not specified

Decision Index · RAGTruth (hallucination F1) as reported by Cloudflare

Card Cloudflare/clef-omni at 0db1cd2607d76a7bdb2a382f659e7b313079f84b; table Decision Index.

Published table label: RAGTruth (hallucination F1).

Vendor-published observation, distinct from independent and other vendor harnesses. Cloudflare internal Decision Index 0.2.1 and workflow cohorts; ForecastBench values are native published Brier ×100, not accuracy.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-01 · Revision: 0b331204bb13fbd2ca93a64df1956f5b55478ce5 · Published observations; result reuse terms not specified

Decision Index · RouterBench (selected quality) as reported by Cloudflare

Card Cloudflare/clef-omni at 0db1cd2607d76a7bdb2a382f659e7b313079f84b; table Decision Index.

Published table label: RouterBench (selected quality).

Vendor-published observation, distinct from independent and other vendor harnesses. Cloudflare internal Decision Index 0.2.1 and workflow cohorts; ForecastBench values are native published Brier ×100, not accuracy.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-01 · Revision: 0b331204bb13fbd2ca93a64df1956f5b55478ce5 · Published observations; result reuse terms not specified

Decision Index · SATA-Bench (case exact accuracy) as reported by Cloudflare

Card Cloudflare/clef-omni at 0db1cd2607d76a7bdb2a382f659e7b313079f84b; table Decision Index.

Published table label: SATA-Bench (case exact accuracy).

Vendor-published observation, distinct from independent and other vendor harnesses. Cloudflare internal Decision Index 0.2.1 and workflow cohorts; ForecastBench values are native published Brier ×100, not accuracy.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-01 · Revision: 0b331204bb13fbd2ca93a64df1956f5b55478ce5 · Published observations; result reuse terms not specified

Decision Index · SGD/SGD-X (macro-F1) as reported by Cloudflare

Card Cloudflare/clef-omni at 0db1cd2607d76a7bdb2a382f659e7b313079f84b; table Decision Index.

Published table label: SGD/SGD-X (macro-F1).

Vendor-published observation, distinct from independent and other vendor harnesses. Cloudflare internal Decision Index 0.2.1 and workflow cohorts; ForecastBench values are native published Brier ×100, not accuracy.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-01 · Revision: 0b331204bb13fbd2ca93a64df1956f5b55478ce5 · Published observations; result reuse terms not specified

Decision Index · ToolRet (nDCG@10) as reported by Cloudflare

Card Cloudflare/clef-omni at 0db1cd2607d76a7bdb2a382f659e7b313079f84b; table Decision Index.

Published table label: ToolRet (nDCG@10).

Vendor-published observation, distinct from independent and other vendor harnesses. Cloudflare internal Decision Index 0.2.1 and workflow cohorts; ForecastBench values are native published Brier ×100, not accuracy.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-01 · Revision: 0b331204bb13fbd2ca93a64df1956f5b55478ce5 · Published observations; result reuse terms not specified

Decision Index · VAST (macro-F1) as reported by Cloudflare

Card Cloudflare/clef-omni at 0db1cd2607d76a7bdb2a382f659e7b313079f84b; table Decision Index.

Published table label: VAST (macro-F1).

Vendor-published observation, distinct from independent and other vendor harnesses. Cloudflare internal Decision Index 0.2.1 and workflow cohorts; ForecastBench values are native published Brier ×100, not accuracy.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-01 · Revision: 0b331204bb13fbd2ca93a64df1956f5b55478ce5 · Published observations; result reuse terms not specified

Decision Index · When2Call MCQ (accuracy) as reported by Cloudflare

Card Cloudflare/clef-omni at 0db1cd2607d76a7bdb2a382f659e7b313079f84b; table Decision Index.

Published table label: When2Call MCQ (accuracy).

Vendor-published observation, distinct from independent and other vendor harnesses. Cloudflare internal Decision Index 0.2.1 and workflow cohorts; ForecastBench values are native published Brier ×100, not accuracy.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-01 · Revision: 0b331204bb13fbd2ca93a64df1956f5b55478ce5 · Published observations; result reuse terms not specified

Decision Index · WinoGrande (accuracy) as reported by Cloudflare

Card Cloudflare/clef-omni at 0db1cd2607d76a7bdb2a382f659e7b313079f84b; table Decision Index.

Published table label: WinoGrande (accuracy).

Vendor-published observation, distinct from independent and other vendor harnesses. Cloudflare internal Decision Index 0.2.1 and workflow cohorts; ForecastBench values are native published Brier ×100, not accuracy.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-01 · Revision: 0b331204bb13fbd2ca93a64df1956f5b55478ce5 · Published observations; result reuse terms not specified

Workflow evals · Agent trace observability · Primary action as reported by Cloudflare

Card Cloudflare/clef-omni at 0db1cd2607d76a7bdb2a382f659e7b313079f84b; table Workflow evals.

Published table label: Agent trace observability · Primary action.

Vendor-published observation, distinct from independent and other vendor harnesses. Cloudflare internal Decision Index 0.2.1 and workflow cohorts; ForecastBench values are native published Brier ×100, not accuracy.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-01 · Revision: 0b331204bb13fbd2ca93a64df1956f5b55478ce5 · Published observations; result reuse terms not specified

Workflow evals · Customer service · Exact actions as reported by Cloudflare

Card Cloudflare/clef-omni at 0db1cd2607d76a7bdb2a382f659e7b313079f84b; table Workflow evals.

Published table label: Customer service · Exact actions.

Vendor-published observation, distinct from independent and other vendor harnesses. Cloudflare internal Decision Index 0.2.1 and workflow cohorts; ForecastBench values are native published Brier ×100, not accuracy.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-01 · Revision: 0b331204bb13fbd2ca93a64df1956f5b55478ce5 · Published observations; result reuse terms not specified

Workflow evals · Invoice processing · Exact actions as reported by Cloudflare

Card Cloudflare/clef-omni at 0db1cd2607d76a7bdb2a382f659e7b313079f84b; table Workflow evals.

Published table label: Invoice processing · Exact actions.

Vendor-published observation, distinct from independent and other vendor harnesses. Cloudflare internal Decision Index 0.2.1 and workflow cohorts; ForecastBench values are native published Brier ×100, not accuracy.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-01 · Revision: 0b331204bb13fbd2ca93a64df1956f5b55478ce5 · Published observations; result reuse terms not specified

Workflow evals · Invoice processing · Primary action as reported by Cloudflare

Card Cloudflare/clef-omni at 0db1cd2607d76a7bdb2a382f659e7b313079f84b; table Workflow evals.

Published table label: Invoice processing · Primary action.

Vendor-published observation, distinct from independent and other vendor harnesses. Cloudflare internal Decision Index 0.2.1 and workflow cohorts; ForecastBench values are native published Brier ×100, not accuracy.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-01 · Revision: 0b331204bb13fbd2ca93a64df1956f5b55478ce5 · Published observations; result reuse terms not specified

Workflow evals · Security incidents · Exact actions as reported by Cloudflare

Card Cloudflare/clef-omni at 0db1cd2607d76a7bdb2a382f659e7b313079f84b; table Workflow evals.

Published table label: Security incidents · Exact actions.

Vendor-published observation, distinct from independent and other vendor harnesses. Cloudflare internal Decision Index 0.2.1 and workflow cohorts; ForecastBench values are native published Brier ×100, not accuracy.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-01 · Revision: 0b331204bb13fbd2ca93a64df1956f5b55478ce5 · Published observations; result reuse terms not specified

Exact named hosted API observation; All-case accuracy.

Named source route cloudflare/clef-omni; ModelTested runner v1; tested_on=openrouter. Exact backend weight revision and API inference precision unpublished.

Publisher generated 2026-10-18. Execution dates, token usage and hashes remain in its per-run receipts. No private prompt/response archive imported.

56 authored situations, including 19 hard cases, each with two typed questions; 14 public and 42 hidden cases. Source aggregate averages per-case decision accuracy. Confidence score is 100×(1−mean multiclass Brier/2), not ECE or a calibration probability. Small single-run benchmark; current HF access/size does not inherit this evaluator’s stale closed/API-only metadata. No tariff inferred from rounded cost 0.0.

Checked / reported: 2026-10-18 · Samples: 56

Snapshot: 2026-10-18 · Revision: 081e8f37130a4a9077b8cffe62827d4c3b9e3b23 · Published aggregate observations; result reuse terms not separately specified

Exact named hosted API observation; Hard-case accuracy.

Named source route cloudflare/clef-omni; ModelTested runner v1; tested_on=openrouter. Exact backend weight revision and API inference precision unpublished.

Publisher generated 2026-10-18. Execution dates, token usage and hashes remain in its per-run receipts. No private prompt/response archive imported.

56 authored situations, including 19 hard cases, each with two typed questions; 14 public and 42 hidden cases. Source aggregate averages per-case decision accuracy. Confidence score is 100×(1−mean multiclass Brier/2), not ECE or a calibration probability. Small single-run benchmark; current HF access/size does not inherit this evaluator’s stale closed/API-only metadata. No tariff inferred from rounded cost 0.0.

Checked / reported: 2026-10-18 · Samples: 19

Snapshot: 2026-10-18 · Revision: 081e8f37130a4a9077b8cffe62827d4c3b9e3b23 · Published aggregate observations; result reuse terms not separately specified

Exact named hosted API observation; Public-case accuracy.

Named source route cloudflare/clef-omni; ModelTested runner v1; tested_on=openrouter. Exact backend weight revision and API inference precision unpublished.

Publisher generated 2026-10-18. Execution dates, token usage and hashes remain in its per-run receipts. No private prompt/response archive imported.

56 authored situations, including 19 hard cases, each with two typed questions; 14 public and 42 hidden cases. Source aggregate averages per-case decision accuracy. Confidence score is 100×(1−mean multiclass Brier/2), not ECE or a calibration probability. Small single-run benchmark; current HF access/size does not inherit this evaluator’s stale closed/API-only metadata. No tariff inferred from rounded cost 0.0.

Checked / reported: 2026-10-18 · Samples: 14

Snapshot: 2026-10-18 · Revision: 081e8f37130a4a9077b8cffe62827d4c3b9e3b23 · Published aggregate observations; result reuse terms not separately specified

Exact named hosted API observation; Hidden-case accuracy.

Named source route cloudflare/clef-omni; ModelTested runner v1; tested_on=openrouter. Exact backend weight revision and API inference precision unpublished.

Publisher generated 2026-10-18. Execution dates, token usage and hashes remain in its per-run receipts. No private prompt/response archive imported.

56 authored situations, including 19 hard cases, each with two typed questions; 14 public and 42 hidden cases. Source aggregate averages per-case decision accuracy. Confidence score is 100×(1−mean multiclass Brier/2), not ECE or a calibration probability. Small single-run benchmark; current HF access/size does not inherit this evaluator’s stale closed/API-only metadata. No tariff inferred from rounded cost 0.0.

Checked / reported: 2026-10-18 · Samples: 42

Snapshot: 2026-10-18 · Revision: 081e8f37130a4a9077b8cffe62827d4c3b9e3b23 · Published aggregate observations; result reuse terms not separately specified

Exact named hosted API observation; Brier-based confidence score.

Named source route cloudflare/clef-omni; ModelTested runner v1; tested_on=openrouter. Exact backend weight revision and API inference precision unpublished.

Publisher generated 2026-10-18. Execution dates, token usage and hashes remain in its per-run receipts. No private prompt/response archive imported.

56 authored situations, including 19 hard cases, each with two typed questions; 14 public and 42 hidden cases. Source aggregate averages per-case decision accuracy. Confidence score is 100×(1−mean multiclass Brier/2), not ECE or a calibration probability. Small single-run benchmark; current HF access/size does not inherit this evaluator’s stale closed/API-only metadata. No tariff inferred from rounded cost 0.0.

Checked / reported: 2026-10-18 · Samples: 56

Snapshot: 2026-10-18 · Revision: 081e8f37130a4a9077b8cffe62827d4c3b9e3b23 · Published aggregate observations; result reuse terms not separately specified

Accuracy under third-party joint-request scorer, distinct from original hosted d1/submitter-specific ECE conventions.

Exact named Cloudflare/clef-omni; zero-shot HF Jobs, vLLM nightly /v1/systemone, --max-model-len 8192 --max-num-seqs 8; one request per case, 400 cases with all 5 questions each; hardware varies and latency is not ranked. Exact weights SHA and vLLM revision unspecified by source summary.

Third-party SaylorTwift HF Jobs public evaluation, 16 October 2026; accepted into publisher card 19–20 October. Test: 400 cases / 2000 typed decisions; all five questions with the state in one /v1/systemone request. Argmax accuracy counts ties in gold as correct; KL(gold||prediction) clips predictions at 1e-9; native multiclass Brier; ECE over 10 equal-width confidence bins. Same public scorer across these 10 model rows; not rerun by dataset owner. This scorer differs from the older hosted d1 row, so separate dataset IDs. Heterogeneous GPU latency excluded. Teacher-produced soft labels define agreement, not objective correctness. Model checkpoint revision and exact vLLM nightly build are not published in the summary; exact named HF checkpoint, no sibling transfers. Submission: https://huggingface.co/datasets/LocalLLaMA/typed-decisions/discussions/37

Third-party SaylorTwift HF Jobs public evaluation, 16 October 2026; accepted into publisher card 19–20 October. Test: 400 cases / 2000 typed decisions; all five questions with the state in one /v1/systemone request. Argmax accuracy counts ties in gold as correct; KL(gold||prediction) clips predictions at 1e-9; native multiclass Brier; ECE over 10 equal-width confidence bins. Same public scorer across these 10 model rows; not rerun by dataset owner. This scorer differs from the older hosted d1 row, so separate dataset IDs. Heterogeneous GPU latency excluded. Teacher-produced soft labels define agreement, not objective correctness. Model checkpoint revision and exact vLLM nightly build are not published in the summary; exact named HF checkpoint, no sibling transfers.

Checked / reported: 2026-10-20 · Evaluated: 2026-10-16 · Samples: 2000

Snapshot: 2026-10-19T22:54:08.000Z · Revision: d9eda7f31ba227333213a1787f2c78fdaa4090c8 · Apache-2.0 dataset/card; third-party published observations

KL from gold under third-party joint-request scorer, distinct from original hosted d1/submitter-specific ECE conventions.

Exact named Cloudflare/clef-omni; zero-shot HF Jobs, vLLM nightly /v1/systemone, --max-model-len 8192 --max-num-seqs 8; one request per case, 400 cases with all 5 questions each; hardware varies and latency is not ranked. Exact weights SHA and vLLM revision unspecified by source summary.

Third-party SaylorTwift HF Jobs public evaluation, 16 October 2026; accepted into publisher card 19–20 October. Test: 400 cases / 2000 typed decisions; all five questions with the state in one /v1/systemone request. Argmax accuracy counts ties in gold as correct; KL(gold||prediction) clips predictions at 1e-9; native multiclass Brier; ECE over 10 equal-width confidence bins. Same public scorer across these 10 model rows; not rerun by dataset owner. This scorer differs from the older hosted d1 row, so separate dataset IDs. Heterogeneous GPU latency excluded. Teacher-produced soft labels define agreement, not objective correctness. Model checkpoint revision and exact vLLM nightly build are not published in the summary; exact named HF checkpoint, no sibling transfers. Submission: https://huggingface.co/datasets/LocalLLaMA/typed-decisions/discussions/37

Third-party SaylorTwift HF Jobs public evaluation, 16 October 2026; accepted into publisher card 19–20 October. Test: 400 cases / 2000 typed decisions; all five questions with the state in one /v1/systemone request. Argmax accuracy counts ties in gold as correct; KL(gold||prediction) clips predictions at 1e-9; native multiclass Brier; ECE over 10 equal-width confidence bins. Same public scorer across these 10 model rows; not rerun by dataset owner. This scorer differs from the older hosted d1 row, so separate dataset IDs. Heterogeneous GPU latency excluded. Teacher-produced soft labels define agreement, not objective correctness. Model checkpoint revision and exact vLLM nightly build are not published in the summary; exact named HF checkpoint, no sibling transfers.

Checked / reported: 2026-10-20 · Evaluated: 2026-10-16 · Samples: 2000

Snapshot: 2026-10-19T22:54:08.000Z · Revision: d9eda7f31ba227333213a1787f2c78fdaa4090c8 · Apache-2.0 dataset/card; third-party published observations

Brier under third-party joint-request scorer, distinct from original hosted d1/submitter-specific ECE conventions.

Exact named Cloudflare/clef-omni; zero-shot HF Jobs, vLLM nightly /v1/systemone, --max-model-len 8192 --max-num-seqs 8; one request per case, 400 cases with all 5 questions each; hardware varies and latency is not ranked. Exact weights SHA and vLLM revision unspecified by source summary.

Third-party SaylorTwift HF Jobs public evaluation, 16 October 2026; accepted into publisher card 19–20 October. Test: 400 cases / 2000 typed decisions; all five questions with the state in one /v1/systemone request. Argmax accuracy counts ties in gold as correct; KL(gold||prediction) clips predictions at 1e-9; native multiclass Brier; ECE over 10 equal-width confidence bins. Same public scorer across these 10 model rows; not rerun by dataset owner. This scorer differs from the older hosted d1 row, so separate dataset IDs. Heterogeneous GPU latency excluded. Teacher-produced soft labels define agreement, not objective correctness. Model checkpoint revision and exact vLLM nightly build are not published in the summary; exact named HF checkpoint, no sibling transfers. Submission: https://huggingface.co/datasets/LocalLLaMA/typed-decisions/discussions/37

Third-party SaylorTwift HF Jobs public evaluation, 16 October 2026; accepted into publisher card 19–20 October. Test: 400 cases / 2000 typed decisions; all five questions with the state in one /v1/systemone request. Argmax accuracy counts ties in gold as correct; KL(gold||prediction) clips predictions at 1e-9; native multiclass Brier; ECE over 10 equal-width confidence bins. Same public scorer across these 10 model rows; not rerun by dataset owner. This scorer differs from the older hosted d1 row, so separate dataset IDs. Heterogeneous GPU latency excluded. Teacher-produced soft labels define agreement, not objective correctness. Model checkpoint revision and exact vLLM nightly build are not published in the summary; exact named HF checkpoint, no sibling transfers.

Checked / reported: 2026-10-20 · Evaluated: 2026-10-16 · Samples: 2000

Snapshot: 2026-10-19T22:54:08.000Z · Revision: d9eda7f31ba227333213a1787f2c78fdaa4090c8 · Apache-2.0 dataset/card; third-party published observations

ECE under third-party joint-request scorer, distinct from original hosted d1/submitter-specific ECE conventions.

Exact named Cloudflare/clef-omni; zero-shot HF Jobs, vLLM nightly /v1/systemone, --max-model-len 8192 --max-num-seqs 8; one request per case, 400 cases with all 5 questions each; hardware varies and latency is not ranked. Exact weights SHA and vLLM revision unspecified by source summary.

Third-party SaylorTwift HF Jobs public evaluation, 16 October 2026; accepted into publisher card 19–20 October. Test: 400 cases / 2000 typed decisions; all five questions with the state in one /v1/systemone request. Argmax accuracy counts ties in gold as correct; KL(gold||prediction) clips predictions at 1e-9; native multiclass Brier; ECE over 10 equal-width confidence bins. Same public scorer across these 10 model rows; not rerun by dataset owner. This scorer differs from the older hosted d1 row, so separate dataset IDs. Heterogeneous GPU latency excluded. Teacher-produced soft labels define agreement, not objective correctness. Model checkpoint revision and exact vLLM nightly build are not published in the summary; exact named HF checkpoint, no sibling transfers. Submission: https://huggingface.co/datasets/LocalLLaMA/typed-decisions/discussions/37

Third-party SaylorTwift HF Jobs public evaluation, 16 October 2026; accepted into publisher card 19–20 October. Test: 400 cases / 2000 typed decisions; all five questions with the state in one /v1/systemone request. Argmax accuracy counts ties in gold as correct; KL(gold||prediction) clips predictions at 1e-9; native multiclass Brier; ECE over 10 equal-width confidence bins. Same public scorer across these 10 model rows; not rerun by dataset owner. This scorer differs from the older hosted d1 row, so separate dataset IDs. Heterogeneous GPU latency excluded. Teacher-produced soft labels define agreement, not objective correctness. Model checkpoint revision and exact vLLM nightly build are not published in the summary; exact named HF checkpoint, no sibling transfers.

Checked / reported: 2026-10-20 · Evaluated: 2026-10-16 · Samples: 2000

Snapshot: 2026-10-19T22:54:08.000Z · Revision: d9eda7f31ba227333213a1787f2c78fdaa4090c8 · Apache-2.0 dataset/card; third-party published observations