Examenos

Four equally weighted evaluations: AA-Omniscience-Decision, HLE-MultiChoice (513 text-only questions), When2Call and WildGuardTest.

Exact named AA decision system; index v1.0. Served precision and detailed execution date not published in inspected chart.

Rounded chart label manually verified in source; not the ordinary language-model Intelligence Index.

Independent AA results. Omniscience splits its 25% between accuracy and non-hallucination. Displayed integer chart values, not undisclosed precise values. No automated AA import or refresh adapter.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-17 · Revision: Manually checked chart · 2026-10-17 · Published observations; result reuse terms not specified

Cloudflare self-reported public Decision Index 0.2.1; not the current Full board or the AA index.

Cloudflare vendor launch chart; self-reported run. Card subtests and later board revisions remain separate.

Printed scatter-plot score labels visually inspected through the public linked image route. Unlabeled latency coordinates are not reconstructed.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-01 · Revision: Launch chart 2026-10-01 · Published observations; result reuse terms not specified

400 contrastive decisions in 200 pairs across eight workflow tasks; pair accuracy needs both answers correct.

{"model_requested": "clef", "model_served": "clef", "endpoint": "AI Space /v1/systemone (Cloudflare Workers AI)", "protocol": "few-shot: one example per option", "client_region": null, "concurrency": 4, "warmup_calls": 3, "harness_commit": "decidebench@4f99337"}

Run 2026-10-09. Workload cost is per 1M tasks, not a token tariff.

Synthetic Claude-written items, structural validation and label spot checks. Few-shot examples except explicit zero-shot column. Hosted costs use observed billed usage and dated tariffs. Hosted latency includes provider/gateway, four concurrent calls and three warmups. Luna timing client differs from the other hosted runs, so latency is not a controlled speed ranking.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-09 · Samples: 400

Snapshot: 2026-10-17 · Revision: bbf8f2c77b544952b42be555866729a22c33980d · CC BY 4.0 original dataset; MIT code/report

400 contrastive decisions in 200 pairs across eight workflow tasks; pair accuracy needs both answers correct.

{"model_requested": "clef", "model_served": "clef", "endpoint": "AI Space /v1/systemone (Cloudflare Workers AI)", "protocol": "few-shot: one example per option", "client_region": null, "concurrency": 4, "warmup_calls": 3, "harness_commit": "decidebench@4f99337"}

Run 2026-10-09. Workload cost is per 1M tasks, not a token tariff.

Synthetic Claude-written items, structural validation and label spot checks. Few-shot examples except explicit zero-shot column. Hosted costs use observed billed usage and dated tariffs. Hosted latency includes provider/gateway, four concurrent calls and three warmups. Luna timing client differs from the other hosted runs, so latency is not a controlled speed ranking.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-09 · Samples: 50

Snapshot: 2026-10-17 · Revision: bbf8f2c77b544952b42be555866729a22c33980d · CC BY 4.0 original dataset; MIT code/report

400 contrastive decisions in 200 pairs across eight workflow tasks; pair accuracy needs both answers correct.

{"model_requested": "clef", "model_served": "clef", "endpoint": "AI Space /v1/systemone (Cloudflare Workers AI)", "protocol": "few-shot: one example per option", "client_region": null, "concurrency": 4, "warmup_calls": 3, "harness_commit": "decidebench@4f99337"}

Run 2026-10-09. Workload cost is per 1M tasks, not a token tariff.

Synthetic Claude-written items, structural validation and label spot checks. Few-shot examples except explicit zero-shot column. Hosted costs use observed billed usage and dated tariffs. Hosted latency includes provider/gateway, four concurrent calls and three warmups. Luna timing client differs from the other hosted runs, so latency is not a controlled speed ranking.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-09 · Samples: 50

Snapshot: 2026-10-17 · Revision: bbf8f2c77b544952b42be555866729a22c33980d · CC BY 4.0 original dataset; MIT code/report

400 contrastive decisions in 200 pairs across eight workflow tasks; pair accuracy needs both answers correct.

{"model_requested": "clef", "model_served": "clef", "endpoint": "AI Space /v1/systemone (Cloudflare Workers AI)", "protocol": "few-shot: one example per option", "client_region": null, "concurrency": 4, "warmup_calls": 3, "harness_commit": "decidebench@4f99337"}

Run 2026-10-09. Workload cost is per 1M tasks, not a token tariff.

Synthetic Claude-written items, structural validation and label spot checks. Few-shot examples except explicit zero-shot column. Hosted costs use observed billed usage and dated tariffs. Hosted latency includes provider/gateway, four concurrent calls and three warmups. Luna timing client differs from the other hosted runs, so latency is not a controlled speed ranking.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-09 · Samples: 50

Snapshot: 2026-10-17 · Revision: bbf8f2c77b544952b42be555866729a22c33980d · CC BY 4.0 original dataset; MIT code/report

400 contrastive decisions in 200 pairs across eight workflow tasks; pair accuracy needs both answers correct.

{"model_requested": "clef", "model_served": "clef", "endpoint": "AI Space /v1/systemone (Cloudflare Workers AI)", "protocol": "few-shot: one example per option", "client_region": null, "concurrency": 4, "warmup_calls": 3, "harness_commit": "decidebench@4f99337"}

Run 2026-10-09. Workload cost is per 1M tasks, not a token tariff.

Synthetic Claude-written items, structural validation and label spot checks. Few-shot examples except explicit zero-shot column. Hosted costs use observed billed usage and dated tariffs. Hosted latency includes provider/gateway, four concurrent calls and three warmups. Luna timing client differs from the other hosted runs, so latency is not a controlled speed ranking.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-09 · Samples: 50

Snapshot: 2026-10-17 · Revision: bbf8f2c77b544952b42be555866729a22c33980d · CC BY 4.0 original dataset; MIT code/report

400 contrastive decisions in 200 pairs across eight workflow tasks; pair accuracy needs both answers correct.

{"model_requested": "clef", "model_served": "clef", "endpoint": "AI Space /v1/systemone (Cloudflare Workers AI)", "protocol": "few-shot: one example per option", "client_region": null, "concurrency": 4, "warmup_calls": 3, "harness_commit": "decidebench@4f99337"}

Run 2026-10-09. Workload cost is per 1M tasks, not a token tariff.

Synthetic Claude-written items, structural validation and label spot checks. Few-shot examples except explicit zero-shot column. Hosted costs use observed billed usage and dated tariffs. Hosted latency includes provider/gateway, four concurrent calls and three warmups. Luna timing client differs from the other hosted runs, so latency is not a controlled speed ranking.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-09 · Samples: 400

Snapshot: 2026-10-17 · Revision: bbf8f2c77b544952b42be555866729a22c33980d · CC BY 4.0 original dataset; MIT code/report

400 contrastive decisions in 200 pairs across eight workflow tasks; pair accuracy needs both answers correct.

{"model_requested": "clef", "model_served": "clef", "endpoint": "AI Space /v1/systemone (Cloudflare Workers AI)", "protocol": "few-shot: one example per option", "client_region": null, "concurrency": 4, "warmup_calls": 3, "harness_commit": "decidebench@4f99337"}

Run 2026-10-09. Workload cost is per 1M tasks, not a token tariff.

Synthetic Claude-written items, structural validation and label spot checks. Few-shot examples except explicit zero-shot column. Hosted costs use observed billed usage and dated tariffs. Hosted latency includes provider/gateway, four concurrent calls and three warmups. Luna timing client differs from the other hosted runs, so latency is not a controlled speed ranking.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-09 · Samples: 400

Snapshot: 2026-10-17 · Revision: bbf8f2c77b544952b42be555866729a22c33980d · CC BY 4.0 original dataset; MIT code/report

400 contrastive decisions in 200 pairs across eight workflow tasks; pair accuracy needs both answers correct.

{"model_requested": "clef", "model_served": "clef", "endpoint": "AI Space /v1/systemone (Cloudflare Workers AI)", "protocol": "few-shot: one example per option", "client_region": null, "concurrency": 4, "warmup_calls": 3, "harness_commit": "decidebench@4f99337"}

Run 2026-10-09. Workload cost is per 1M tasks, not a token tariff.

Synthetic Claude-written items, structural validation and label spot checks. Few-shot examples except explicit zero-shot column. Hosted costs use observed billed usage and dated tariffs. Hosted latency includes provider/gateway, four concurrent calls and three warmups. Luna timing client differs from the other hosted runs, so latency is not a controlled speed ranking.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-09 · Samples: 200

Snapshot: 2026-10-17 · Revision: bbf8f2c77b544952b42be555866729a22c33980d · CC BY 4.0 original dataset; MIT code/report

400 contrastive decisions in 200 pairs across eight workflow tasks; pair accuracy needs both answers correct.

{"model_requested": "clef", "model_served": "clef", "endpoint": "AI Space /v1/systemone (Cloudflare Workers AI)", "protocol": "few-shot: one example per option", "client_region": null, "concurrency": 4, "warmup_calls": 3, "harness_commit": "decidebench@4f99337"}

Run 2026-10-09. Workload cost is per 1M tasks, not a token tariff.

Synthetic Claude-written items, structural validation and label spot checks. Few-shot examples except explicit zero-shot column. Hosted costs use observed billed usage and dated tariffs. Hosted latency includes provider/gateway, four concurrent calls and three warmups. Luna timing client differs from the other hosted runs, so latency is not a controlled speed ranking.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-09 · Samples: 50

Snapshot: 2026-10-17 · Revision: bbf8f2c77b544952b42be555866729a22c33980d · CC BY 4.0 original dataset; MIT code/report

400 contrastive decisions in 200 pairs across eight workflow tasks; pair accuracy needs both answers correct.

{"model_requested": "clef", "model_served": "clef", "endpoint": "AI Space /v1/systemone (Cloudflare Workers AI)", "protocol": "few-shot: one example per option", "client_region": null, "concurrency": 4, "warmup_calls": 3, "harness_commit": "decidebench@4f99337"}

Run 2026-10-09. Workload cost is per 1M tasks, not a token tariff.

Synthetic Claude-written items, structural validation and label spot checks. Few-shot examples except explicit zero-shot column. Hosted costs use observed billed usage and dated tariffs. Hosted latency includes provider/gateway, four concurrent calls and three warmups. Luna timing client differs from the other hosted runs, so latency is not a controlled speed ranking.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-09 · Samples: 50

Snapshot: 2026-10-17 · Revision: bbf8f2c77b544952b42be555866729a22c33980d · CC BY 4.0 original dataset; MIT code/report

400 contrastive decisions in 200 pairs across eight workflow tasks; pair accuracy needs both answers correct.

{"model_requested": "clef", "model_served": "clef", "endpoint": "AI Space /v1/systemone (Cloudflare Workers AI)", "protocol": "few-shot: one example per option", "client_region": null, "concurrency": 4, "warmup_calls": 3, "harness_commit": "decidebench@4f99337"}

Run 2026-10-09. Workload cost is per 1M tasks, not a token tariff.

Synthetic Claude-written items, structural validation and label spot checks. Few-shot examples except explicit zero-shot column. Hosted costs use observed billed usage and dated tariffs. Hosted latency includes provider/gateway, four concurrent calls and three warmups. Luna timing client differs from the other hosted runs, so latency is not a controlled speed ranking.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-09 · Samples: 50

Snapshot: 2026-10-17 · Revision: bbf8f2c77b544952b42be555866729a22c33980d · CC BY 4.0 original dataset; MIT code/report

400 contrastive decisions in 200 pairs across eight workflow tasks; pair accuracy needs both answers correct.

{"model_requested": "clef", "model_served": "clef", "endpoint": "AI Space /v1/systemone (Cloudflare Workers AI)", "protocol": "few-shot: one example per option", "client_region": null, "concurrency": 4, "warmup_calls": 3, "harness_commit": "decidebench@4f99337"}

Run 2026-10-09. Workload cost is per 1M tasks, not a token tariff.

Synthetic Claude-written items, structural validation and label spot checks. Few-shot examples except explicit zero-shot column. Hosted costs use observed billed usage and dated tariffs. Hosted latency includes provider/gateway, four concurrent calls and three warmups. Luna timing client differs from the other hosted runs, so latency is not a controlled speed ranking.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-09 · Samples: 50

Snapshot: 2026-10-17 · Revision: bbf8f2c77b544952b42be555866729a22c33980d · CC BY 4.0 original dataset; MIT code/report

BFCL formulated as typed decisions; published case exact accuracy for this board population.

Board identity clef; evaluator run. Public model-card serving path; actual revision and precision not otherwise established.

Raw 0.9847; {"requests": 1694, "answered": 1694, "unsupported": 0, "errors": 0}; coverage/error rules are source-defined.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: 1694

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

SGD/SGD-X formulated as typed decisions; published macro-F1 for this board population.

Board identity clef; evaluator run. Public model-card serving path; actual revision and precision not otherwise established.

Raw 0.4381; {"requests": 2500, "answered": 2500, "unsupported": 0, "errors": 0}; coverage/error rules are source-defined.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: 2500

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

ContractNLI formulated as typed decisions; published macro-F1 for this board population.

Board identity clef; evaluator run. Public model-card serving path; actual revision and precision not otherwise established.

Raw 0.8135; {"requests": 123, "answered": 123, "unsupported": 0, "errors": 0}; coverage/error rules are source-defined.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: 123

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

ANLI formulated as typed decisions; published macro-F1 for this board population.

Board identity clef; evaluator run. Public model-card serving path; actual revision and precision not otherwise established.

Raw 0.6997; {"requests": 3200, "answered": 3200, "unsupported": 0, "errors": 0}; coverage/error rules are source-defined.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: 3200

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

ToolRet formulated as typed decisions; published nDCG@10 for this board population.

Board identity clef; evaluator run. Public model-card serving path; actual revision and precision not otherwise established.

Raw 0.6906; {"requests": 685, "answered": 685, "unsupported": 0, "errors": 0}; coverage/error rules are source-defined.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: 685

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

BPoMP formulated as typed decisions; published accuracy for this board population.

Board identity clef; evaluator run. Public model-card serving path; actual revision and precision not otherwise established.

Raw 0.9698; {"requests": 5000, "answered": 5000, "unsupported": 0, "errors": 0}; coverage/error rules are source-defined.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: 5000

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

Humicroedit formulated as typed decisions; published accuracy for this board population.

Board identity clef; evaluator run. Public model-card serving path; actual revision and precision not otherwise established.

Raw 0.6655; {"requests": 2628, "answered": 2628, "unsupported": 0, "errors": 0}; coverage/error rules are source-defined.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: 2628

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

POP909-CL formulated as typed decisions; published accuracy for this board population.

Board identity clef; evaluator run. Public model-card serving path; actual revision and precision not otherwise established.

Raw 0.159; {"requests": 2000, "answered": 2000, "unsupported": 0, "errors": 0}; coverage/error rules are source-defined.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: 2000

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

cfcolor formulated as typed decisions; published accuracy for this board population.

Board identity clef; evaluator run. Public model-card serving path; actual revision and precision not otherwise established.

Raw 0.6594; {"requests": 5000, "answered": 5000, "unsupported": 0, "errors": 0}; coverage/error rules are source-defined.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: 5000

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

MMLU formulated as typed decisions; published accuracy for this board population.

Board identity clef; evaluator run. Public model-card serving path; actual revision and precision not otherwise established.

Raw 0.9022; {"requests": 14033, "answered": 14033, "unsupported": 0, "errors": 0}; coverage/error rules are source-defined.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: 14033

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

GPQA Diamond formulated as typed decisions; published accuracy for this board population.

Board identity clef; evaluator run. Public model-card serving path; actual revision and precision not otherwise established.

Raw 0.4847; {"requests": 196, "answered": 196, "unsupported": 0, "errors": 0}; coverage/error rules are source-defined.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: 196

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

ARC-Easy formulated as typed decisions; published accuracy for this board population.

Board identity clef; evaluator run. Public model-card serving path; actual revision and precision not otherwise established.

Raw 0.9903; {"requests": 2376, "answered": 2376, "unsupported": 0, "errors": 0}; coverage/error rules are source-defined.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: 2376

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

ARC-Challenge formulated as typed decisions; published accuracy for this board population.

Board identity clef; evaluator run. Public model-card serving path; actual revision and precision not otherwise established.

Raw 0.9778; {"requests": 1172, "answered": 1172, "unsupported": 0, "errors": 0}; coverage/error rules are source-defined.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: 1172

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

WinoGrande formulated as typed decisions; published accuracy for this board population.

Board identity clef; evaluator run. Public model-card serving path; actual revision and precision not otherwise established.

Raw 0.9353; {"requests": 1267, "answered": 1267, "unsupported": 0, "errors": 0}; coverage/error rules are source-defined.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: 1267

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

HellaSwag formulated as typed decisions; published accuracy for this board population.

Board identity clef; evaluator run. Public model-card serving path; actual revision and precision not otherwise established.

Raw 0.9818; {"requests": 10042, "answered": 10042, "unsupported": 0, "errors": 0}; coverage/error rules are source-defined.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: 10042

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

API-Bank formulated as typed decisions; published accuracy for this board population.

Board identity clef; evaluator run. Public model-card serving path; actual revision and precision not otherwise established.

Raw 0.9193; {"requests": 508, "answered": 508, "unsupported": 0, "errors": 0}; coverage/error rules are source-defined.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: 508

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

GSM8K formulated as typed decisions; published accuracy for this board population.

Board identity clef; evaluator run. Public model-card serving path; actual revision and precision not otherwise established.

Raw 0.5993; {"requests": 2638, "answered": 2638, "unsupported": 0, "errors": 0}; coverage/error rules are source-defined.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: 2638

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

ChessBench formulated as typed decisions; published accuracy for this board population.

Board identity clef; evaluator run. Public model-card serving path; actual revision and precision not otherwise established.

Raw 0.245; {"requests": 5000, "answered": 5000, "unsupported": 0, "errors": 0}; coverage/error rules are source-defined.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: 5000

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

MuSR formulated as typed decisions; published accuracy for this board population.

Board identity clef; evaluator run. Public model-card serving path; actual revision and precision not otherwise established.

Raw 0.8378; {"requests": 752, "answered": 752, "unsupported": 0, "errors": 0}; coverage/error rules are source-defined.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: 752

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

SATA-Bench formulated as typed decisions; published case exact accuracy for this board population.

Board identity clef; evaluator run. Public model-card serving path; actual revision and precision not otherwise established.

Raw 0.3345; {"requests": 1650, "answered": 1650, "unsupported": 0, "errors": 0}; coverage/error rules are source-defined.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: 1650

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

BRIGHT formulated as typed decisions; published nDCG@10 for this board population.

Board identity clef; evaluator run. Public model-card serving path; actual revision and precision not otherwise established.

Raw 0.4746; {"requests": 220, "answered": 220, "unsupported": 0, "errors": 0}; coverage/error rules are source-defined.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: 220

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

Amazon ESCI formulated as typed decisions; published macro-F1 for this board population.

Board identity clef; evaluator run. Public model-card serving path; actual revision and precision not otherwise established.

Raw 0.5737; {"requests": 5000, "answered": 5000, "unsupported": 0, "errors": 0}; coverage/error rules are source-defined.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: 5000

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

ACOS formulated as typed decisions; published per-review F1 for this board population.

Board identity clef; evaluator run. Public model-card serving path; actual revision and precision not otherwise established.

Raw 0.3317; {"requests": 1565, "answered": 1565, "unsupported": 0, "errors": 0}; coverage/error rules are source-defined.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: 1565

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

FinEntity formulated as typed decisions; published macro-F1 for this board population.

Board identity clef; evaluator run. Public model-card serving path; actual revision and precision not otherwise established.

Raw 0.9609; {"requests": 979, "answered": 979, "unsupported": 0, "errors": 0}; coverage/error rules are source-defined.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: 979

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

BANKING77 formulated as typed decisions; published macro-F1 for this board population.

Board identity clef; evaluator run. Public model-card serving path; actual revision and precision not otherwise established.

Raw 0.9408; {"requests": 3080, "answered": 3080, "unsupported": 0, "errors": 0}; coverage/error rules are source-defined.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: 3080

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

iSarcasmEval formulated as typed decisions; published Sarcasm F1 · track A, English for this board population.

Board identity clef; evaluator run. Public model-card serving path; actual revision and precision not otherwise established.

Raw 0.4639; {"requests": 4600, "answered": 4600, "unsupported": 0, "errors": 0}; coverage/error rules are source-defined.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: 4600

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

VAST formulated as typed decisions; published macro-F1 for this board population.

Board identity clef; evaluator run. Public model-card serving path; actual revision and precision not otherwise established.

Raw 0.5934; {"requests": 3006, "answered": 3006, "unsupported": 0, "errors": 0}; coverage/error rules are source-defined.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: 3006

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

NLI4CT formulated as typed decisions; published macro-F1 for this board population.

Board identity clef; evaluator run. Public model-card serving path; actual revision and precision not otherwise established.

Raw 0.8298; {"requests": 5500, "answered": 5500, "unsupported": 0, "errors": 0}; coverage/error rules are source-defined.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: 5500

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

CRUXEval formulated as typed decisions; published accuracy for this board population.

Board identity clef; evaluator run. Public model-card serving path; actual revision and precision not otherwise established.

Raw 0.8649; {"requests": 570, "answered": 570, "unsupported": 0, "errors": 0}; coverage/error rules are source-defined.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: 570

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

CLadder formulated as typed decisions; published accuracy for this board population.

Board identity clef; evaluator run. Public model-card serving path; actual revision and precision not otherwise established.

Raw 0.9408; {"requests": 5000, "answered": 5000, "unsupported": 0, "errors": 0}; coverage/error rules are source-defined.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: 5000

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

HLE formulated as typed decisions; published accuracy for this board population.

Board identity clef; evaluator run. Public model-card serving path; actual revision and precision not otherwise established.

Raw 0.1277; {"requests": 501, "answered": 501, "unsupported": 0, "errors": 0}; coverage/error rules are source-defined.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: 501

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

CLINC150+OOS formulated as typed decisions; published macro-F1 for this board population.

Board identity clef; evaluator run. Public model-card serving path; actual revision and precision not otherwise established.

Raw 0.9741; {"requests": 5500, "answered": 5500, "unsupported": 0, "errors": 0}; coverage/error rules are source-defined.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: 5500

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

Habermas Machine formulated as typed decisions; published accuracy for this board population.

Board identity clef; evaluator run. Public model-card serving path; actual revision and precision not otherwise established.

Raw 0.6868; {"requests": 1676, "answered": 1676, "unsupported": 0, "errors": 0}; coverage/error rules are source-defined.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: 1676

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

PhishNChips phishing decisions formulated as typed decisions; published accuracy for this board population.

Board identity clef; evaluator run. Public model-card serving path; actual revision and precision not otherwise established.

Raw 0.7935; {"requests": 2000, "answered": 2000, "unsupported": 0, "errors": 0}; coverage/error rules are source-defined.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: 2000

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

MMLU-Pro formulated as typed decisions; published accuracy for this board population.

Board identity clef; evaluator run. Public model-card serving path; actual revision and precision not otherwise established.

Raw 0.6597; {"requests": 12032, "answered": 12032, "unsupported": 0, "errors": 0}; coverage/error rules are source-defined.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: 12032

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

BBH formulated as typed decisions; published accuracy for this board population.

Board identity clef; evaluator run. Public model-card serving path; actual revision and precision not otherwise established.

Raw 0.7363; {"requests": 5507, "answered": 5507, "unsupported": 0, "errors": 0}; coverage/error rules are source-defined.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: 5507

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

RAGTruth formulated as typed decisions; published F1 on hallucinated class for this board population.

Board identity clef; evaluator run. Public model-card serving path; actual revision and precision not otherwise established.

Raw 0.7954; {"requests": 2700, "answered": 2700, "unsupported": 0, "errors": 0}; coverage/error rules are source-defined.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: 2700

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

RouterBench formulated as typed decisions; published selected quality (quality objective) for this board population.

Board identity clef; evaluator run. Public model-card serving path; actual revision and precision not otherwise established.

Raw 0.7966; {"requests": 10000, "answered": 10000, "unsupported": 0, "errors": 0}; coverage/error rules are source-defined.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: 10000

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

HoVer formulated as typed decisions; published accuracy for this board population.

Board identity clef; evaluator run. Public model-card serving path; actual revision and precision not otherwise established.

Raw 0.6558; {"requests": 4000, "answered": 4000, "unsupported": 0, "errors": 0}; coverage/error rules are source-defined.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: 4000

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

When2Call formulated as typed decisions; published accuracy for this board population.

Board identity clef; evaluator run. Public model-card serving path; actual revision and precision not otherwise established.

Raw 0.7245; {"requests": 3652, "answered": 3652, "unsupported": 0, "errors": 0}; coverage/error rules are source-defined.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: 3652

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

New Yorker formulated as typed decisions; published accuracy for this board population.

Board identity clef; evaluator run. Public model-card serving path; actual revision and precision not otherwise established.

Raw 0.7027; {"requests": 528, "answered": 528, "unsupported": 0, "errors": 0}; coverage/error rules are source-defined.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: 528

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

Home appliance simulator formulated as typed decisions; published case exact accuracy for this board population.

Board identity clef; evaluator run. Public model-card serving path; actual revision and precision not otherwise established.

Raw 0.8068; {"requests": 88, "answered": 88, "unsupported": 0, "errors": 0}; coverage/error rules are source-defined.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: 88

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

Calibration on the source-defined text sample; confidence alone is not a quality ranking.

Board identity clef; evaluator run. Public model-card serving path; actual revision and precision not otherwise established.

Raw acc: 0.7783; 32 benchmarks. ECE uses the publisher bins and weighting; native Brier convention stays distinct from other studies.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported. Uses roughly one sixth of requests with discrete correct answers; benchmarks receive equal weight. ToolRet, BRIGHT and RouterBench are excluded. Known training exposure is penalized.

Checked / reported: 2026-10-17 · Samples: 35088

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

Calibration on the source-defined text sample; confidence alone is not a quality ranking.

Board identity clef; evaluator run. Public model-card serving path; actual revision and precision not otherwise established.

Raw brier: 0.306; 32 benchmarks. ECE uses the publisher bins and weighting; native Brier convention stays distinct from other studies.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported. Uses roughly one sixth of requests with discrete correct answers; benchmarks receive equal weight. ToolRet, BRIGHT and RouterBench are excluded. Known training exposure is penalized.

Checked / reported: 2026-10-17 · Samples: 35088

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

Calibration on the source-defined text sample; confidence alone is not a quality ranking.

Board identity clef; evaluator run. Public model-card serving path; actual revision and precision not otherwise established.

Raw conf: 0.8096; 32 benchmarks. ECE uses the publisher bins and weighting; native Brier convention stays distinct from other studies.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported. Uses roughly one sixth of requests with discrete correct answers; benchmarks receive equal weight. ToolRet, BRIGHT and RouterBench are excluded. Known training exposure is penalized.

Checked / reported: 2026-10-17 · Samples: 35088

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

Calibration on the source-defined text sample; confidence alone is not a quality ranking.

Board identity clef; evaluator run. Public model-card serving path; actual revision and precision not otherwise established.

Raw ece: 0.0395; 32 benchmarks. ECE uses the publisher bins and weighting; native Brier convention stays distinct from other studies.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported. Uses roughly one sixth of requests with discrete correct answers; benchmarks receive equal weight. ToolRet, BRIGHT and RouterBench are excluded. Known training exposure is penalized.

Checked / reported: 2026-10-17 · Samples: 35088

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

Published Median latency for the board setup; hosted network and local inference are separate.

Board identity clef; evaluator run. Public model-card serving path; actual revision and precision not otherwise established.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported. Local GPU/runtime can differ between submitted and evaluator runs; not isolated architecture speed.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

Text Decision Index 0.3.2 · Private new-domain score

Board identity clef; evaluator run. Public model-card serving path; actual revision and precision not otherwise established.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

Text Decision Index 0.3.2 · Public score

Board identity clef; evaluator run. Public model-card serving path; actual revision and precision not otherwise established.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

Text Decision Index 0.3.2 · Private same-skill score

Board identity clef; evaluator run. Public model-card serving path; actual revision and precision not otherwise established.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

Text Decision Index 0.3.2 · Full score

Board identity clef; evaluator run. Public model-card serving path; actual revision and precision not otherwise established.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

JevBench Binary slices: aegis2 · acc @0.5

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 83.2%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: aegis2 · acc if flipped

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 16.8%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: aegis2 · AUROC

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.9226.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: aegis2 · Brier

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.1235.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: aegis2 · ECE

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.0620.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: aegis2 · FPR

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 22.0%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: aegis2 · positive recall

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 88.4%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: aegis2_response · acc @0.5

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 78.6%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: aegis2_response · acc if flipped

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 21.4%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: aegis2_response · AUROC

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.8912.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: aegis2_response · Brier

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.1498.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: aegis2_response · ECE

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.0489.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: aegis2_response · FPR

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 35.2%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: aegis2_response · positive recall

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 92.4%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: atbench500 · acc @0.5

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 93.8%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: atbench500 · acc if flipped

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 6.2%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: atbench500 · AUROC

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.9860.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: atbench500 · Brier

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.0580.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: atbench500 · ECE

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.0801.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: atbench500 · FPR

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 11.2%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: atbench500 · positive recall

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 98.8%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: jailbreak_classification · acc @0.5

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 97.0%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 400

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: jailbreak_classification · acc if flipped

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 3.0%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 400

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: jailbreak_classification · AUROC

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.9977.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 400

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: jailbreak_classification · Brier

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.0211.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 400

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: jailbreak_classification · ECE

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.0158.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 400

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: jailbreak_classification · FPR

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.0%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 400

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: jailbreak_classification · positive recall

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 91.5%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 400

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: prompt_injections · acc @0.5

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 80.2%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 116

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: prompt_injections · acc if flipped

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 19.8%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 116

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: prompt_injections · AUROC

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.9943.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 116

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: prompt_injections · Brier

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.1417.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 116

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: prompt_injections · ECE

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.1140.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 116

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: prompt_injections · FPR

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.0%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 116

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: prompt_injections · positive recall

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 61.7%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 116

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench By subset: rewardbench2 · accuracy

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 67.9%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 240

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench By subset: rewardbench2 · accuracy

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 70.0%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 240

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench By subset: rewardbench2 · accuracy

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 75.0%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 240

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench By subset: rewardbench2 · accuracy

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 32.9%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 240

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench By subset: rewardbench2 · accuracy

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 88.8%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 240

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: banking77 · accuracy

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 94.8%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 1000

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: banking77 · Brier

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.0802.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 1000

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: banking77 · ECE (10 bins)

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.0428.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 1000

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: banking77 · macro-F1

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.9465.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 1000

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: banking77 · NLL

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.2066.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 1000

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: mcpbench_server_routing · accuracy

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 95.5%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 112

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: mcpbench_server_routing · Brier

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.0590.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 112

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: mcpbench_server_routing · ECE (10 bins)

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.1084.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 112

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: mcpbench_server_routing · macro-F1

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.9415.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 112

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: mcpbench_server_routing · NLL

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.1728.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 112

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: mcpbench_tool_selection · accuracy

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 100.0%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 26

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: mcpbench_tool_selection · Brier

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.0216.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 26

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: mcpbench_tool_selection · ECE (10 bins)

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.0917.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 26

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: mcpbench_tool_selection · macro-F1

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 1.0000.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 26

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: mcpbench_tool_selection · NLL

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.1020.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 26

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: mcpmark_server_routing · accuracy

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 55.0%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 338

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: mcpmark_server_routing · Brier

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.6189.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 338

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: mcpmark_server_routing · ECE (10 bins)

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.1915.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 338

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: mcpmark_server_routing · macro-F1

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.4886.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 338

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: mcpmark_server_routing · NLL

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 1.2868.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 338

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: mcpmark_tool_selection · accuracy

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 100.0%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 18

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: mcpmark_tool_selection · Brier

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.0008.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 18

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: mcpmark_tool_selection · ECE (10 bins)

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.0213.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 18

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: mcpmark_tool_selection · macro-F1

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 1.0000.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 18

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: mcpmark_tool_selection · NLL

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.0216.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 18

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: medmcqa · accuracy

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 72.5%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 1000

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: medmcqa · Brier

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.3733.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 1000

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: medmcqa · ECE (10 bins)

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.0656.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 1000

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: medmcqa · macro-F1

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.7253.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 1000

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: medmcqa · NLL

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.7004.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 1000

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: medqa_usmle · accuracy

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 84.3%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 1000

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: medqa_usmle · Brier

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.2199.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 1000

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: medqa_usmle · ECE (10 bins)

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.0364.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 1000

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: medqa_usmle · macro-F1

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.8427.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 1000

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: medqa_usmle · NLL

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.4220.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 1000

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: mmlu_pro · accuracy

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 66.8%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 1000

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: mmlu_pro · Brier

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.4323.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 1000

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: mmlu_pro · ECE (10 bins)

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.0363.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 1000

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: mmlu_pro · macro-F1

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.6644.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 1000

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: mmlu_pro · NLL

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 1.0117.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 1000

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: pubmedqa · accuracy

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 67.8%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: pubmedqa · Brier

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.4579.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: pubmedqa · ECE (10 bins)

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.1001.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: pubmedqa · macro-F1

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.5819.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: pubmedqa · NLL

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.8138.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: rewardbench2 · accuracy

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 66.9%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 1200

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: rewardbench2 · Brier

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.4792.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 1200

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: rewardbench2 · ECE (10 bins)

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.1379.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 1200

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: rewardbench2 · macro-F1

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.6689.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 1200

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: rewardbench2 · NLL

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.9085.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 1200

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: scienceqa_text · accuracy

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 93.9%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 1000

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: scienceqa_text · Brier

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.0945.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 1000

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: scienceqa_text · ECE (10 bins)

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.0096.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 1000

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: scienceqa_text · macro-F1

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.9593.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 1000

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: scienceqa_text · NLL

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.1682.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 1000

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Headline: aegis2 · accuracy

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.8320.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 500 · Source 95% CI: 79.60000000000001–86.4

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Headline: aegis2_response · accuracy

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.7860.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 500 · Source 95% CI: 75–82.19999999999999

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Headline: atbench500 · accuracy

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.9380.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 500 · Source 95% CI: 91.60000000000001–95.8

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Headline: banking77 · accuracy

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.9480.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 1000 · Source 95% CI: 92.9–96.7

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Headline: helpsteer2_helpfulness · mae_expected_level

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.9343.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 500 · Source 95% CI: 0.877–0.997

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Headline: jailbreak_classification · accuracy

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.9700.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 400 · Source 95% CI: 95.3–98.5

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Headline: mcpbench_server_routing · server_routing_accuracy

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.9554.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 112 · Source 95% CI: 90.2–100

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Headline: mcpbench_tool_selection · tool_selection_accuracy

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 1.0000.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 26

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Headline: mcpmark_server_routing · server_routing_accuracy

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.5503.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 338 · Source 95% CI: 47.3–62.7

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Headline: mcpmark_tool_selection · tool_selection_accuracy

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 1.0000.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 18

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Headline: medmcqa · accuracy

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.7250.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 1000 · Source 95% CI: 68.8–76.1

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Headline: medqa_usmle · accuracy

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.8430.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 1000 · Source 95% CI: 81.2–87.2

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Headline: mmlu_pro · accuracy

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.6680.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 1000 · Source 95% CI: 63.1–70.8

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Headline: nfcorpus_rank_choice · ndcg@10

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.3692.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 298 · Source 95% CI: 33.5–40.6

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Headline: nfcorpus_rank_graded · ndcg@10

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.3764.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 4924 · Source 95% CI: 34.2–41.3

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Headline: nfcorpus_rank_noul · ndcg@10

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.3732.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 298 · Source 95% CI: 33.7–41

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Headline: prompt_injections · accuracy

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.8017.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 116 · Source 95% CI: 72.39999999999999–87.1

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Headline: pubmedqa · accuracy

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.6780.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 500 · Source 95% CI: 63.6–71.8

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Headline: rewardbench2 · accuracy

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.6692.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 1200 · Source 95% CI: 63.4–70.1

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Headline: scienceqa_text · accuracy

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.9390.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 1000 · Source 95% CI: 92.10000000000001–95.7

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Headline: scifact_rank_choice · ndcg@10

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.7520.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 300 · Source 95% CI: 70.7–79.5

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Headline: scifact_rank_noul · ndcg@10

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.7448.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 300 · Source 95% CI: 70–78.8

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Headline: sst5 · mae_expected_level

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.5359.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 500 · Source 95% CI: 0.502–0.572

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: banking77 · both correct

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 94.6%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: banking77 · flipped by reordering

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.4%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: banking77 · original order

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 94.6%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: banking77 · permuted order

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 95.0%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: mcpbench_server_routing · both correct

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 94.6%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 56

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: mcpbench_server_routing · flipped by reordering

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 1.8%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 56

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: mcpbench_server_routing · original order

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 94.6%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 56

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: mcpbench_server_routing · permuted order

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 96.4%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 56

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: mcpbench_tool_selection · both correct

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 100.0%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 4

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: mcpbench_tool_selection · flipped by reordering

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.0%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 4

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: mcpbench_tool_selection · original order

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 100.0%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 4

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: mcpbench_tool_selection · permuted order

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 100.0%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 4

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: mcpmark_server_routing · both correct

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 54.4%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 169

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: mcpmark_server_routing · flipped by reordering

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 1.2%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 169

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: mcpmark_server_routing · original order

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 54.4%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 169

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: mcpmark_server_routing · permuted order

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 55.6%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 169

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: mcpmark_tool_selection · both correct

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 100.0%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 3

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: mcpmark_tool_selection · flipped by reordering

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.0%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 3

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: mcpmark_tool_selection · original order

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 100.0%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 3

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: mcpmark_tool_selection · permuted order

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 100.0%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 3

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: medmcqa · both correct

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 66.0%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: medmcqa · flipped by reordering

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 13.0%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: medmcqa · original order

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 72.0%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: medmcqa · permuted order

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 73.0%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: medqa_usmle · both correct

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 80.8%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: medqa_usmle · flipped by reordering

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 7.0%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: medqa_usmle · original order

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 84.8%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: medqa_usmle · permuted order

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 83.8%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: mmlu_pro · both correct

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 60.8%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: mmlu_pro · flipped by reordering

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 12.0%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: mmlu_pro · original order

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 67.4%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: mmlu_pro · permuted order

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 66.2%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: rewardbench2 · both correct

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 58.1%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 594

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: rewardbench2 · flipped by reordering

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 17.7%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 594

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: rewardbench2 · original order

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 67.2%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 594

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: rewardbench2 · permuted order

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 66.7%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 594

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: scienceqa_text · both correct

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 91.4%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: scienceqa_text · flipped by reordering

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 5.0%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: scienceqa_text · original order

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 94.2%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: scienceqa_text · permuted order

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 93.6%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ordinal slices: helpsteer2_helpfulness · exact (rounded)

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 32.0%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ordinal slices: helpsteer2_helpfulness · MAE (expected level)

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.934.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ordinal slices: helpsteer2_helpfulness · RPS

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.1617.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ordinal slices: helpsteer2_helpfulness · Spearman

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.510.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ordinal slices: helpsteer2_helpfulness · within 1 level

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 80.0%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ordinal slices: nfcorpus_rank_graded · exact (rounded)

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 63.0%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 4924

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ordinal slices: nfcorpus_rank_graded · MAE (expected level)

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.464.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 4924

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ordinal slices: nfcorpus_rank_graded · RPS

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.1362.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 4924

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ordinal slices: nfcorpus_rank_graded · Spearman

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.486.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 4924

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ordinal slices: nfcorpus_rank_graded · within 1 level

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 96.5%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 4924

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ordinal slices: sst5 · exact (rounded)

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 53.6%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ordinal slices: sst5 · MAE (expected level)

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.536.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ordinal slices: sst5 · RPS

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.0831.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ordinal slices: sst5 · Spearman

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.883.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ordinal slices: sst5 · within 1 level

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 97.6%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ranking slices: nfcorpus_rank_choice · MRR@10

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.6199.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 298

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ranking slices: nfcorpus_rank_choice · nDCG@10

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.3692.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 298

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ranking slices: nfcorpus_rank_choice · Recall@10

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.1705.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 298

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ranking slices: nfcorpus_rank_graded · MRR@10

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.6177.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 298

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ranking slices: nfcorpus_rank_graded · nDCG@10

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.3764.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 298

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ranking slices: nfcorpus_rank_graded · Recall@10

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.1749.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 298

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ranking slices: nfcorpus_rank_noul · MRR@10

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.6166.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 298

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ranking slices: nfcorpus_rank_noul · nDCG@10

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.3732.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 298

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ranking slices: nfcorpus_rank_noul · Recall@10

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.1727.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 298

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ranking slices: scifact_rank_choice · MRR@10

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.7376.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 300

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ranking slices: scifact_rank_choice · nDCG@10

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.7520.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 300

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ranking slices: scifact_rank_choice · Recall@10

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.8251.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 300

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ranking slices: scifact_rank_noul · MRR@10

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.7293.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 300

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ranking slices: scifact_rank_noul · nDCG@10

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.7448.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 300

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ranking slices: scifact_rank_noul · Recall@10

Scored 2026-10-15T23:38:30-0700. Model requested `or-cloudflare-clef`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.8217.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:38:30-0700 · Samples: 300

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

Liquid vendor’s 112 unique evaluations grouped as Average; not an independent index.

Liquid vendor-run October 15 panel. Exact d1-ultra/Jev 1.14/d3 labels; remaining endpoint revisions and served precision unspecified.

Figure-only printed quantity. Source does not publish per-task weights, complete item populations or all inference settings. Average is the vendor’s published aggregate, never recomputed from rounded domains. Peer model exact hosted revisions can be unspecified.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-15 · Revision: Release figures 2026-10-15 · Published observations; result reuse terms not specified

Liquid vendor’s 112 unique evaluations grouped as Business and new fields; not an independent index.

Liquid vendor-run October 15 panel. Exact d1-ultra/Jev 1.14/d3 labels; remaining endpoint revisions and served precision unspecified.

Figure-only printed quantity. Source does not publish per-task weights, complete item populations or all inference settings. Average is the vendor’s published aggregate, never recomputed from rounded domains. Peer model exact hosted revisions can be unspecified.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-15 · Revision: Release figures 2026-10-15 · Published observations; result reuse terms not specified

Liquid vendor’s 112 unique evaluations grouped as Judgment and taste; not an independent index.

Liquid vendor-run October 15 panel. Exact d1-ultra/Jev 1.14/d3 labels; remaining endpoint revisions and served precision unspecified.

Figure-only printed quantity. Source does not publish per-task weights, complete item populations or all inference settings. Average is the vendor’s published aggregate, never recomputed from rounded domains. Peer model exact hosted revisions can be unspecified.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-15 · Revision: Release figures 2026-10-15 · Published observations; result reuse terms not specified

Liquid vendor’s 112 unique evaluations grouped as Knowledge and reasoning; not an independent index.

Liquid vendor-run October 15 panel. Exact d1-ultra/Jev 1.14/d3 labels; remaining endpoint revisions and served precision unspecified.

Figure-only printed quantity. Source does not publish per-task weights, complete item populations or all inference settings. Average is the vendor’s published aggregate, never recomputed from rounded domains. Peer model exact hosted revisions can be unspecified.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-15 · Revision: Release figures 2026-10-15 · Published observations; result reuse terms not specified

Liquid vendor’s 112 unique evaluations grouped as Languages; not an independent index.

Liquid vendor-run October 15 panel. Exact d1-ultra/Jev 1.14/d3 labels; remaining endpoint revisions and served precision unspecified.

Figure-only printed quantity. Source does not publish per-task weights, complete item populations or all inference settings. Average is the vendor’s published aggregate, never recomputed from rounded domains. Peer model exact hosted revisions can be unspecified.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-15 · Revision: Release figures 2026-10-15 · Published observations; result reuse terms not specified

Liquid vendor’s 112 unique evaluations grouped as Pictures; not an independent index.

Liquid vendor-run October 15 panel. Exact d1-ultra/Jev 1.14/d3 labels; remaining endpoint revisions and served precision unspecified.

Figure-only printed quantity. Source does not publish per-task weights, complete item populations or all inference settings. Average is the vendor’s published aggregate, never recomputed from rounded domains. Peer model exact hosted revisions can be unspecified.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-15 · Revision: Release figures 2026-10-15 · Published observations; result reuse terms not specified

Liquid vendor’s 112 unique evaluations grouped as Robustness; not an independent index.

Liquid vendor-run October 15 panel. Exact d1-ultra/Jev 1.14/d3 labels; remaining endpoint revisions and served precision unspecified.

Figure-only printed quantity. Source does not publish per-task weights, complete item populations or all inference settings. Average is the vendor’s published aggregate, never recomputed from rounded domains. Peer model exact hosted revisions can be unspecified.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-15 · Revision: Release figures 2026-10-15 · Published observations; result reuse terms not specified

Liquid vendor’s 112 unique evaluations grouped as Safety and policy; not an independent index.

Liquid vendor-run October 15 panel. Exact d1-ultra/Jev 1.14/d3 labels; remaining endpoint revisions and served precision unspecified.

Figure-only printed quantity. Source does not publish per-task weights, complete item populations or all inference settings. Average is the vendor’s published aggregate, never recomputed from rounded domains. Peer model exact hosted revisions can be unspecified.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-15 · Revision: Release figures 2026-10-15 · Published observations; result reuse terms not specified

Liquid vendor’s 112 unique evaluations grouped as Tools and agents; not an independent index.

Liquid vendor-run October 15 panel. Exact d1-ultra/Jev 1.14/d3 labels; remaining endpoint revisions and served precision unspecified.

Figure-only printed quantity. Source does not publish per-task weights, complete item populations or all inference settings. Average is the vendor’s published aggregate, never recomputed from rounded domains. Peer model exact hosted revisions can be unspecified.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-15 · Revision: Release figures 2026-10-15 · Published observations; result reuse terms not specified

Vendor-defined mean of absolute accuracy minus chosen confidence; not interchangeable with other studies’ ECE.

Liquid vendor figure: mean absolute accuracy−confidence on chosen option; hardware/precision unpublished.

Printed figure values manually inspected. Curve coordinates are not reconstructed. Source task weighting and binning are not fully published.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-15 · Revision: Release figure 2026-10-15 · Published observations; result reuse terms not specified

TypeSafe’s evolving private internal measurement; its own warning says likely biased toward its models.

TypeSafe release figure; exact named checkpoint. Internal suite/hardware and detailed calibration definition unpublished.

Figure-only printed values manually inspected. No stable suite or metric formula is published. Not the AA or HF Decision Index, and not comparable to other ECE definitions.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-14 · Revision: Launch figures 2026-10-14 · Published observations; result reuse terms not specified

TypeSafe’s evolving private internal measurement; its own warning says likely biased toward its models.

TypeSafe release figure; exact named checkpoint. Internal suite/hardware and detailed calibration definition unpublished.

Figure-only printed values manually inspected. No stable suite or metric formula is published. Not the AA or HF Decision Index, and not comparable to other ECE definitions.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-14 · Revision: Launch figures 2026-10-14 · Published observations; result reuse terms not specified

Decision Index · ACOS (per-review F1) as reported by Cloudflare

Card Cloudflare/clef at 0b331204bb13fbd2ca93a64df1956f5b55478ce5; table Decision Index.

Published table label: ACOS (per-review F1).

Vendor-published observation, distinct from independent and other vendor harnesses. Cloudflare internal Decision Index 0.2.1 and workflow cohorts; ForecastBench values are native published Brier ×100, not accuracy.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-01 · Revision: 0b331204bb13fbd2ca93a64df1956f5b55478ce5 · Published observations; result reuse terms not specified

Decision Index · Amazon ESCI (macro-F1) as reported by Cloudflare

Card Cloudflare/clef at 0b331204bb13fbd2ca93a64df1956f5b55478ce5; table Decision Index.

Published table label: Amazon ESCI (macro-F1).

Vendor-published observation, distinct from independent and other vendor harnesses. Cloudflare internal Decision Index 0.2.1 and workflow cohorts; ForecastBench values are native published Brier ×100, not accuracy.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-01 · Revision: 0b331204bb13fbd2ca93a64df1956f5b55478ce5 · Published observations; result reuse terms not specified

Decision Index · ANLI (macro-F1) as reported by Cloudflare

Card Cloudflare/clef at 0b331204bb13fbd2ca93a64df1956f5b55478ce5; table Decision Index.

Published table label: ANLI (macro-F1).

Vendor-published observation, distinct from independent and other vendor harnesses. Cloudflare internal Decision Index 0.2.1 and workflow cohorts; ForecastBench values are native published Brier ×100, not accuracy.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-01 · Revision: 0b331204bb13fbd2ca93a64df1956f5b55478ce5 · Published observations; result reuse terms not specified

Decision Index · API-Bank (accuracy) as reported by Cloudflare

Card Cloudflare/clef at 0b331204bb13fbd2ca93a64df1956f5b55478ce5; table Decision Index.

Published table label: API-Bank (accuracy).

Vendor-published observation, distinct from independent and other vendor harnesses. Cloudflare internal Decision Index 0.2.1 and workflow cohorts; ForecastBench values are native published Brier ×100, not accuracy.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-01 · Revision: 0b331204bb13fbd2ca93a64df1956f5b55478ce5 · Published observations; result reuse terms not specified

Decision Index · ARC-Challenge (accuracy) as reported by Cloudflare

Card Cloudflare/clef at 0b331204bb13fbd2ca93a64df1956f5b55478ce5; table Decision Index.

Published table label: ARC-Challenge (accuracy).

Vendor-published observation, distinct from independent and other vendor harnesses. Cloudflare internal Decision Index 0.2.1 and workflow cohorts; ForecastBench values are native published Brier ×100, not accuracy.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-01 · Revision: 0b331204bb13fbd2ca93a64df1956f5b55478ce5 · Published observations; result reuse terms not specified

Decision Index · ARC-Easy (accuracy) as reported by Cloudflare

Card Cloudflare/clef at 0b331204bb13fbd2ca93a64df1956f5b55478ce5; table Decision Index.

Published table label: ARC-Easy (accuracy).

Vendor-published observation, distinct from independent and other vendor harnesses. Cloudflare internal Decision Index 0.2.1 and workflow cohorts; ForecastBench values are native published Brier ×100, not accuracy.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-01 · Revision: 0b331204bb13fbd2ca93a64df1956f5b55478ce5 · Published observations; result reuse terms not specified

Decision Index · BANKING77 (macro-F1) as reported by Cloudflare

Card Cloudflare/clef at 0b331204bb13fbd2ca93a64df1956f5b55478ce5; table Decision Index.

Published table label: BANKING77 (macro-F1).

Vendor-published observation, distinct from independent and other vendor harnesses. Cloudflare internal Decision Index 0.2.1 and workflow cohorts; ForecastBench values are native published Brier ×100, not accuracy.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-01 · Revision: 0b331204bb13fbd2ca93a64df1956f5b55478ce5 · Published observations; result reuse terms not specified

Decision Index · BBH (accuracy) as reported by Cloudflare

Card Cloudflare/clef at 0b331204bb13fbd2ca93a64df1956f5b55478ce5; table Decision Index.

Published table label: BBH (accuracy).

Vendor-published observation, distinct from independent and other vendor harnesses. Cloudflare internal Decision Index 0.2.1 and workflow cohorts; ForecastBench values are native published Brier ×100, not accuracy.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-01 · Revision: 0b331204bb13fbd2ca93a64df1956f5b55478ce5 · Published observations; result reuse terms not specified

Decision Index · BFCL (case exact accuracy) as reported by Cloudflare

Card Cloudflare/clef at 0b331204bb13fbd2ca93a64df1956f5b55478ce5; table Decision Index.

Published table label: BFCL (case exact accuracy).

Vendor-published observation, distinct from independent and other vendor harnesses. Cloudflare internal Decision Index 0.2.1 and workflow cohorts; ForecastBench values are native published Brier ×100, not accuracy.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-01 · Revision: 0b331204bb13fbd2ca93a64df1956f5b55478ce5 · Published observations; result reuse terms not specified

Decision Index · BPoMP (accuracy) as reported by Cloudflare

Card Cloudflare/clef at 0b331204bb13fbd2ca93a64df1956f5b55478ce5; table Decision Index.

Published table label: BPoMP (accuracy).

Vendor-published observation, distinct from independent and other vendor harnesses. Cloudflare internal Decision Index 0.2.1 and workflow cohorts; ForecastBench values are native published Brier ×100, not accuracy.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-01 · Revision: 0b331204bb13fbd2ca93a64df1956f5b55478ce5 · Published observations; result reuse terms not specified

Decision Index · BRIGHT (nDCG@10) as reported by Cloudflare

Card Cloudflare/clef at 0b331204bb13fbd2ca93a64df1956f5b55478ce5; table Decision Index.

Published table label: BRIGHT (nDCG@10).

Vendor-published observation, distinct from independent and other vendor harnesses. Cloudflare internal Decision Index 0.2.1 and workflow cohorts; ForecastBench values are native published Brier ×100, not accuracy.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-01 · Revision: 0b331204bb13fbd2ca93a64df1956f5b55478ce5 · Published observations; result reuse terms not specified

Decision Index · cfcolor (accuracy) as reported by Cloudflare

Card Cloudflare/clef at 0b331204bb13fbd2ca93a64df1956f5b55478ce5; table Decision Index.

Published table label: cfcolor (accuracy).

Vendor-published observation, distinct from independent and other vendor harnesses. Cloudflare internal Decision Index 0.2.1 and workflow cohorts; ForecastBench values are native published Brier ×100, not accuracy.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-01 · Revision: 0b331204bb13fbd2ca93a64df1956f5b55478ce5 · Published observations; result reuse terms not specified

Decision Index · ChessBench (accuracy) as reported by Cloudflare

Card Cloudflare/clef at 0b331204bb13fbd2ca93a64df1956f5b55478ce5; table Decision Index.

Published table label: ChessBench (accuracy).

Vendor-published observation, distinct from independent and other vendor harnesses. Cloudflare internal Decision Index 0.2.1 and workflow cohorts; ForecastBench values are native published Brier ×100, not accuracy.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-01 · Revision: 0b331204bb13fbd2ca93a64df1956f5b55478ce5 · Published observations; result reuse terms not specified

Decision Index · CLadder (accuracy) as reported by Cloudflare

Card Cloudflare/clef at 0b331204bb13fbd2ca93a64df1956f5b55478ce5; table Decision Index.

Published table label: CLadder (accuracy).

Vendor-published observation, distinct from independent and other vendor harnesses. Cloudflare internal Decision Index 0.2.1 and workflow cohorts; ForecastBench values are native published Brier ×100, not accuracy.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-01 · Revision: 0b331204bb13fbd2ca93a64df1956f5b55478ce5 · Published observations; result reuse terms not specified

Decision Index · CLINC150+OOS (macro-F1) as reported by Cloudflare

Card Cloudflare/clef at 0b331204bb13fbd2ca93a64df1956f5b55478ce5; table Decision Index.

Published table label: CLINC150+OOS (macro-F1).

Vendor-published observation, distinct from independent and other vendor harnesses. Cloudflare internal Decision Index 0.2.1 and workflow cohorts; ForecastBench values are native published Brier ×100, not accuracy.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-01 · Revision: 0b331204bb13fbd2ca93a64df1956f5b55478ce5 · Published observations; result reuse terms not specified

Decision Index · ContractNLI (macro-F1) as reported by Cloudflare

Card Cloudflare/clef at 0b331204bb13fbd2ca93a64df1956f5b55478ce5; table Decision Index.

Published table label: ContractNLI (macro-F1).

Vendor-published observation, distinct from independent and other vendor harnesses. Cloudflare internal Decision Index 0.2.1 and workflow cohorts; ForecastBench values are native published Brier ×100, not accuracy.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-01 · Revision: 0b331204bb13fbd2ca93a64df1956f5b55478ce5 · Published observations; result reuse terms not specified

Decision Index · CRUXEval (accuracy) as reported by Cloudflare

Card Cloudflare/clef at 0b331204bb13fbd2ca93a64df1956f5b55478ce5; table Decision Index.

Published table label: CRUXEval (accuracy).

Vendor-published observation, distinct from independent and other vendor harnesses. Cloudflare internal Decision Index 0.2.1 and workflow cohorts; ForecastBench values are native published Brier ×100, not accuracy.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-01 · Revision: 0b331204bb13fbd2ca93a64df1956f5b55478ce5 · Published observations; result reuse terms not specified

Decision Index · FinEntity (macro-F1) as reported by Cloudflare

Card Cloudflare/clef at 0b331204bb13fbd2ca93a64df1956f5b55478ce5; table Decision Index.

Published table label: FinEntity (macro-F1).

Vendor-published observation, distinct from independent and other vendor harnesses. Cloudflare internal Decision Index 0.2.1 and workflow cohorts; ForecastBench values are native published Brier ×100, not accuracy.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-01 · Revision: 0b331204bb13fbd2ca93a64df1956f5b55478ce5 · Published observations; result reuse terms not specified

Decision Index · ForecastBench (Brier, lower is better) as reported by Cloudflare

Card Cloudflare/clef at 0b331204bb13fbd2ca93a64df1956f5b55478ce5; table Decision Index.

Published table label: ForecastBench (Brier, lower is better).

Vendor-published observation, distinct from independent and other vendor harnesses. Cloudflare internal Decision Index 0.2.1 and workflow cohorts; ForecastBench values are native published Brier ×100, not accuracy.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-01 · Revision: 0b331204bb13fbd2ca93a64df1956f5b55478ce5 · Published observations; result reuse terms not specified

Decision Index · GPQA Diamond (accuracy) as reported by Cloudflare

Card Cloudflare/clef at 0b331204bb13fbd2ca93a64df1956f5b55478ce5; table Decision Index.

Published table label: GPQA Diamond (accuracy).

Vendor-published observation, distinct from independent and other vendor harnesses. Cloudflare internal Decision Index 0.2.1 and workflow cohorts; ForecastBench values are native published Brier ×100, not accuracy.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-01 · Revision: 0b331204bb13fbd2ca93a64df1956f5b55478ce5 · Published observations; result reuse terms not specified

Decision Index · GSM8K (accuracy) as reported by Cloudflare

Card Cloudflare/clef at 0b331204bb13fbd2ca93a64df1956f5b55478ce5; table Decision Index.

Published table label: GSM8K (accuracy).

Vendor-published observation, distinct from independent and other vendor harnesses. Cloudflare internal Decision Index 0.2.1 and workflow cohorts; ForecastBench values are native published Brier ×100, not accuracy.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-01 · Revision: 0b331204bb13fbd2ca93a64df1956f5b55478ce5 · Published observations; result reuse terms not specified

Decision Index · Habermas Machine (accuracy) as reported by Cloudflare

Card Cloudflare/clef at 0b331204bb13fbd2ca93a64df1956f5b55478ce5; table Decision Index.

Published table label: Habermas Machine (accuracy).

Vendor-published observation, distinct from independent and other vendor harnesses. Cloudflare internal Decision Index 0.2.1 and workflow cohorts; ForecastBench values are native published Brier ×100, not accuracy.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-01 · Revision: 0b331204bb13fbd2ca93a64df1956f5b55478ce5 · Published observations; result reuse terms not specified

Decision Index · HellaSwag (accuracy) as reported by Cloudflare

Card Cloudflare/clef at 0b331204bb13fbd2ca93a64df1956f5b55478ce5; table Decision Index.

Published table label: HellaSwag (accuracy).

Vendor-published observation, distinct from independent and other vendor harnesses. Cloudflare internal Decision Index 0.2.1 and workflow cohorts; ForecastBench values are native published Brier ×100, not accuracy.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-01 · Revision: 0b331204bb13fbd2ca93a64df1956f5b55478ce5 · Published observations; result reuse terms not specified

Decision Index · Home appliance simulator (case exact accuracy) as reported by Cloudflare

Card Cloudflare/clef at 0b331204bb13fbd2ca93a64df1956f5b55478ce5; table Decision Index.

Published table label: Home appliance simulator (case exact accuracy).

Vendor-published observation, distinct from independent and other vendor harnesses. Cloudflare internal Decision Index 0.2.1 and workflow cohorts; ForecastBench values are native published Brier ×100, not accuracy.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-01 · Revision: 0b331204bb13fbd2ca93a64df1956f5b55478ce5 · Published observations; result reuse terms not specified

Decision Index · HoVer (accuracy) as reported by Cloudflare

Card Cloudflare/clef at 0b331204bb13fbd2ca93a64df1956f5b55478ce5; table Decision Index.

Published table label: HoVer (accuracy).

Vendor-published observation, distinct from independent and other vendor harnesses. Cloudflare internal Decision Index 0.2.1 and workflow cohorts; ForecastBench values are native published Brier ×100, not accuracy.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-01 · Revision: 0b331204bb13fbd2ca93a64df1956f5b55478ce5 · Published observations; result reuse terms not specified

Decision Index · Humicroedit (accuracy) as reported by Cloudflare

Card Cloudflare/clef at 0b331204bb13fbd2ca93a64df1956f5b55478ce5; table Decision Index.

Published table label: Humicroedit (accuracy).

Vendor-published observation, distinct from independent and other vendor harnesses. Cloudflare internal Decision Index 0.2.1 and workflow cohorts; ForecastBench values are native published Brier ×100, not accuracy.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-01 · Revision: 0b331204bb13fbd2ca93a64df1956f5b55478ce5 · Published observations; result reuse terms not specified

Decision Index · Median latency (ms) as reported by Cloudflare

Card Cloudflare/clef at 0b331204bb13fbd2ca93a64df1956f5b55478ce5; table Decision Index.

Published table label: Median latency (ms).

Vendor-published observation, distinct from independent and other vendor harnesses. Cloudflare internal Decision Index 0.2.1 and workflow cohorts; ForecastBench values are native published Brier ×100, not accuracy.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-01 · Revision: 0b331204bb13fbd2ca93a64df1956f5b55478ce5 · Published observations; result reuse terms not specified

Decision Index · MMLU (accuracy) as reported by Cloudflare

Card Cloudflare/clef at 0b331204bb13fbd2ca93a64df1956f5b55478ce5; table Decision Index.

Published table label: MMLU (accuracy).

Vendor-published observation, distinct from independent and other vendor harnesses. Cloudflare internal Decision Index 0.2.1 and workflow cohorts; ForecastBench values are native published Brier ×100, not accuracy.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-01 · Revision: 0b331204bb13fbd2ca93a64df1956f5b55478ce5 · Published observations; result reuse terms not specified

Decision Index · MMLU-Pro (accuracy) as reported by Cloudflare

Card Cloudflare/clef at 0b331204bb13fbd2ca93a64df1956f5b55478ce5; table Decision Index.

Published table label: MMLU-Pro (accuracy).

Vendor-published observation, distinct from independent and other vendor harnesses. Cloudflare internal Decision Index 0.2.1 and workflow cohorts; ForecastBench values are native published Brier ×100, not accuracy.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-01 · Revision: 0b331204bb13fbd2ca93a64df1956f5b55478ce5 · Published observations; result reuse terms not specified

Decision Index · MuSR (accuracy) as reported by Cloudflare

Card Cloudflare/clef at 0b331204bb13fbd2ca93a64df1956f5b55478ce5; table Decision Index.

Published table label: MuSR (accuracy).

Vendor-published observation, distinct from independent and other vendor harnesses. Cloudflare internal Decision Index 0.2.1 and workflow cohorts; ForecastBench values are native published Brier ×100, not accuracy.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-01 · Revision: 0b331204bb13fbd2ca93a64df1956f5b55478ce5 · Published observations; result reuse terms not specified

Decision Index · New Yorker (accuracy) as reported by Cloudflare

Card Cloudflare/clef at 0b331204bb13fbd2ca93a64df1956f5b55478ce5; table Decision Index.

Published table label: New Yorker (accuracy).

Vendor-published observation, distinct from independent and other vendor harnesses. Cloudflare internal Decision Index 0.2.1 and workflow cohorts; ForecastBench values are native published Brier ×100, not accuracy.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-01 · Revision: 0b331204bb13fbd2ca93a64df1956f5b55478ce5 · Published observations; result reuse terms not specified

Decision Index · NLI4CT (macro-F1) as reported by Cloudflare

Card Cloudflare/clef at 0b331204bb13fbd2ca93a64df1956f5b55478ce5; table Decision Index.

Published table label: NLI4CT (macro-F1).

Vendor-published observation, distinct from independent and other vendor harnesses. Cloudflare internal Decision Index 0.2.1 and workflow cohorts; ForecastBench values are native published Brier ×100, not accuracy.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-01 · Revision: 0b331204bb13fbd2ca93a64df1956f5b55478ce5 · Published observations; result reuse terms not specified

Decision Index · p95 latency (ms) as reported by Cloudflare

Card Cloudflare/clef at 0b331204bb13fbd2ca93a64df1956f5b55478ce5; table Decision Index.

Published table label: p95 latency (ms).

Vendor-published observation, distinct from independent and other vendor harnesses. Cloudflare internal Decision Index 0.2.1 and workflow cohorts; ForecastBench values are native published Brier ×100, not accuracy.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-01 · Revision: 0b331204bb13fbd2ca93a64df1956f5b55478ce5 · Published observations; result reuse terms not specified

Decision Index · PhishNChips (accuracy) as reported by Cloudflare

Card Cloudflare/clef at 0b331204bb13fbd2ca93a64df1956f5b55478ce5; table Decision Index.

Published table label: PhishNChips (accuracy).

Vendor-published observation, distinct from independent and other vendor harnesses. Cloudflare internal Decision Index 0.2.1 and workflow cohorts; ForecastBench values are native published Brier ×100, not accuracy.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-01 · Revision: 0b331204bb13fbd2ca93a64df1956f5b55478ce5 · Published observations; result reuse terms not specified

Decision Index · POP909-CL (accuracy) as reported by Cloudflare

Card Cloudflare/clef at 0b331204bb13fbd2ca93a64df1956f5b55478ce5; table Decision Index.

Published table label: POP909-CL (accuracy).

Vendor-published observation, distinct from independent and other vendor harnesses. Cloudflare internal Decision Index 0.2.1 and workflow cohorts; ForecastBench values are native published Brier ×100, not accuracy.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-01 · Revision: 0b331204bb13fbd2ca93a64df1956f5b55478ce5 · Published observations; result reuse terms not specified

Decision Index · RAGTruth (hallucination F1) as reported by Cloudflare

Card Cloudflare/clef at 0b331204bb13fbd2ca93a64df1956f5b55478ce5; table Decision Index.

Published table label: RAGTruth (hallucination F1).

Vendor-published observation, distinct from independent and other vendor harnesses. Cloudflare internal Decision Index 0.2.1 and workflow cohorts; ForecastBench values are native published Brier ×100, not accuracy.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-01 · Revision: 0b331204bb13fbd2ca93a64df1956f5b55478ce5 · Published observations; result reuse terms not specified

Decision Index · RouterBench (selected quality) as reported by Cloudflare

Card Cloudflare/clef at 0b331204bb13fbd2ca93a64df1956f5b55478ce5; table Decision Index.

Published table label: RouterBench (selected quality).

Vendor-published observation, distinct from independent and other vendor harnesses. Cloudflare internal Decision Index 0.2.1 and workflow cohorts; ForecastBench values are native published Brier ×100, not accuracy.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-01 · Revision: 0b331204bb13fbd2ca93a64df1956f5b55478ce5 · Published observations; result reuse terms not specified

Decision Index · SATA-Bench (case exact accuracy) as reported by Cloudflare

Card Cloudflare/clef at 0b331204bb13fbd2ca93a64df1956f5b55478ce5; table Decision Index.

Published table label: SATA-Bench (case exact accuracy).

Vendor-published observation, distinct from independent and other vendor harnesses. Cloudflare internal Decision Index 0.2.1 and workflow cohorts; ForecastBench values are native published Brier ×100, not accuracy.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-01 · Revision: 0b331204bb13fbd2ca93a64df1956f5b55478ce5 · Published observations; result reuse terms not specified

Decision Index · SGD/SGD-X (macro-F1) as reported by Cloudflare

Card Cloudflare/clef at 0b331204bb13fbd2ca93a64df1956f5b55478ce5; table Decision Index.

Published table label: SGD/SGD-X (macro-F1).

Vendor-published observation, distinct from independent and other vendor harnesses. Cloudflare internal Decision Index 0.2.1 and workflow cohorts; ForecastBench values are native published Brier ×100, not accuracy.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-01 · Revision: 0b331204bb13fbd2ca93a64df1956f5b55478ce5 · Published observations; result reuse terms not specified

Decision Index · ToolRet (nDCG@10) as reported by Cloudflare

Card Cloudflare/clef at 0b331204bb13fbd2ca93a64df1956f5b55478ce5; table Decision Index.

Published table label: ToolRet (nDCG@10).

Vendor-published observation, distinct from independent and other vendor harnesses. Cloudflare internal Decision Index 0.2.1 and workflow cohorts; ForecastBench values are native published Brier ×100, not accuracy.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-01 · Revision: 0b331204bb13fbd2ca93a64df1956f5b55478ce5 · Published observations; result reuse terms not specified

Decision Index · VAST (macro-F1) as reported by Cloudflare

Card Cloudflare/clef at 0b331204bb13fbd2ca93a64df1956f5b55478ce5; table Decision Index.

Published table label: VAST (macro-F1).

Vendor-published observation, distinct from independent and other vendor harnesses. Cloudflare internal Decision Index 0.2.1 and workflow cohorts; ForecastBench values are native published Brier ×100, not accuracy.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-01 · Revision: 0b331204bb13fbd2ca93a64df1956f5b55478ce5 · Published observations; result reuse terms not specified

Decision Index · When2Call MCQ (accuracy) as reported by Cloudflare

Card Cloudflare/clef at 0b331204bb13fbd2ca93a64df1956f5b55478ce5; table Decision Index.

Published table label: When2Call MCQ (accuracy).

Vendor-published observation, distinct from independent and other vendor harnesses. Cloudflare internal Decision Index 0.2.1 and workflow cohorts; ForecastBench values are native published Brier ×100, not accuracy.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-01 · Revision: 0b331204bb13fbd2ca93a64df1956f5b55478ce5 · Published observations; result reuse terms not specified

Decision Index · WinoGrande (accuracy) as reported by Cloudflare

Card Cloudflare/clef at 0b331204bb13fbd2ca93a64df1956f5b55478ce5; table Decision Index.

Published table label: WinoGrande (accuracy).

Vendor-published observation, distinct from independent and other vendor harnesses. Cloudflare internal Decision Index 0.2.1 and workflow cohorts; ForecastBench values are native published Brier ×100, not accuracy.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-01 · Revision: 0b331204bb13fbd2ca93a64df1956f5b55478ce5 · Published observations; result reuse terms not specified

Workflow evals · Agent trace observability · Primary action as reported by Cloudflare

Card Cloudflare/clef at 0b331204bb13fbd2ca93a64df1956f5b55478ce5; table Workflow evals.

Published table label: Agent trace observability · Primary action.

Vendor-published observation, distinct from independent and other vendor harnesses. Cloudflare internal Decision Index 0.2.1 and workflow cohorts; ForecastBench values are native published Brier ×100, not accuracy.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-01 · Revision: 0b331204bb13fbd2ca93a64df1956f5b55478ce5 · Published observations; result reuse terms not specified

Workflow evals · Customer service · Exact actions as reported by Cloudflare

Card Cloudflare/clef at 0b331204bb13fbd2ca93a64df1956f5b55478ce5; table Workflow evals.

Published table label: Customer service · Exact actions.

Vendor-published observation, distinct from independent and other vendor harnesses. Cloudflare internal Decision Index 0.2.1 and workflow cohorts; ForecastBench values are native published Brier ×100, not accuracy.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-01 · Revision: 0b331204bb13fbd2ca93a64df1956f5b55478ce5 · Published observations; result reuse terms not specified

Workflow evals · Invoice processing · Exact actions as reported by Cloudflare

Card Cloudflare/clef at 0b331204bb13fbd2ca93a64df1956f5b55478ce5; table Workflow evals.

Published table label: Invoice processing · Exact actions.

Vendor-published observation, distinct from independent and other vendor harnesses. Cloudflare internal Decision Index 0.2.1 and workflow cohorts; ForecastBench values are native published Brier ×100, not accuracy.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-01 · Revision: 0b331204bb13fbd2ca93a64df1956f5b55478ce5 · Published observations; result reuse terms not specified

Workflow evals · Invoice processing · Primary action as reported by Cloudflare

Card Cloudflare/clef at 0b331204bb13fbd2ca93a64df1956f5b55478ce5; table Workflow evals.

Published table label: Invoice processing · Primary action.

Vendor-published observation, distinct from independent and other vendor harnesses. Cloudflare internal Decision Index 0.2.1 and workflow cohorts; ForecastBench values are native published Brier ×100, not accuracy.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-01 · Revision: 0b331204bb13fbd2ca93a64df1956f5b55478ce5 · Published observations; result reuse terms not specified

Workflow evals · Security incidents · Exact actions as reported by Cloudflare

Card Cloudflare/clef at 0b331204bb13fbd2ca93a64df1956f5b55478ce5; table Workflow evals.

Published table label: Security incidents · Exact actions.

Vendor-published observation, distinct from independent and other vendor harnesses. Cloudflare internal Decision Index 0.2.1 and workflow cohorts; ForecastBench values are native published Brier ×100, not accuracy.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-01 · Revision: 0b331204bb13fbd2ca93a64df1956f5b55478ce5 · Published observations; result reuse terms not specified

OneJev peer figures · Public index

OneJev launch figure, peer-reported 16 October board observation.

Vendor-run Decision Index 0.3 public kit; 37 benchmarks / 140,178 requests as reported. Source quotes peers from the 16 October board. Not AA, evaluator-private Full, or S1MB. Exact execution dates, budgets and individual run variances are not published.

Vendor-run Decision Index 0.3 public kit; 37 benchmarks / 140,178 requests as reported. Source quotes peers from the 16 October board. Not AA, evaluator-private Full, or S1MB. Exact execution dates, budgets and individual run variances are not published.

Checked / reported: 2026-10-18 · Samples: Not published

Snapshot: 2026-10-17 · Revision: OneJev 1.6 launch · 2026-10-17 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/aegis2 / test · binary_brier. Population 100 cases / 148 decisions.

Exact Cloudflare/clef; adapter clef; settings {"questions_per_call": "all-case-questions"}. Run Cloudflare__clef.

Published measured S1MB run; community execution/submission status is not assumed independent. All published diagnostics: {"binary_brier":0.16112712063878493,"accuracy":0.7702702702702703,"positive_recall":0.7183098591549296,"specificity":0.8181818181818182,"precision":0.7846153846153846,"balanced_accuracy":0.7682458386683739,"f1":0.75,"true_positive":51,"false_positive":14,"false_negative":20,"true_negative":63,"positive_prevalence":0.4797297297297297,"majority_accuracy_baseline":0.5202702702702703,"constant_brier_baseline":0.2495891161431699,"brier_skill":0.354430501102625,"baseline_adjusted_skill":0.5364916773367479,"baseline_adjusted_score":0.5364916773367479}. Counts: {"cases": 100, "expected": 148, "succeeded": 148, "failed": 0}. Parameter-count conventions excluding lookup/embedding parameters are not model MoE active counts. No executed weight SHA is stated in this display record; exact named repository and adapter are retained.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-18 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/aqua_rat / test · target_mass_at_prediction. Population 100 cases / 100 decisions.

Exact Cloudflare/clef; adapter clef; settings {"questions_per_call": "all-case-questions"}. Run Cloudflare__clef.

Published measured S1MB run; community execution/submission status is not assumed independent. Choice is target mass at the selected option; soft targets are not ordinary hard-label accuracy. All published diagnostics: {"target_mass_at_prediction":0.64,"uniform_accuracy_baseline":0.20049999999999962,"fixed_answer_accuracy_baseline":0.29,"baseline_adjusted_skill":0.4929577464788733,"baseline_adjusted_score":0.4929577464788733}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0}. Parameter-count conventions excluding lookup/embedding parameters are not model MoE active counts. No executed weight SHA is stated in this display record; exact named repository and adapter are retained.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-18 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/arc / test · target_mass_at_prediction. Population 100 cases / 100 decisions.

Exact Cloudflare/clef; adapter clef; settings {"questions_per_call": "all-case-questions"}. Run Cloudflare__clef.

Published measured S1MB run; community execution/submission status is not assumed independent. Choice is target mass at the selected option; soft targets are not ordinary hard-label accuracy. All published diagnostics: {"target_mass_at_prediction":0.98,"uniform_accuracy_baseline":0.2533333333333333,"fixed_answer_accuracy_baseline":0.32,"baseline_adjusted_skill":0.9705882352941176,"baseline_adjusted_score":0.9705882352941176}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0}. Parameter-count conventions excluding lookup/embedding parameters are not model MoE active counts. No executed weight SHA is stated in this display record; exact named repository and adapter are retained.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-18 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/arct / test · target_mass_at_prediction. Population 100 cases / 100 decisions.

Exact Cloudflare/clef; adapter clef; settings {"questions_per_call": "all-case-questions"}. Run Cloudflare__clef.

Published measured S1MB run; community execution/submission status is not assumed independent. Choice is target mass at the selected option; soft targets are not ordinary hard-label accuracy. All published diagnostics: {"target_mass_at_prediction":0.87,"uniform_accuracy_baseline":0.5,"fixed_answer_accuracy_baseline":0.53,"baseline_adjusted_skill":0.7234042553191489,"baseline_adjusted_score":0.7234042553191489}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0}. Parameter-count conventions excluding lookup/embedding parameters are not model MoE active counts. No executed weight SHA is stated in this display record; exact named repository and adapter are retained.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-18 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/argument_quality / test · target_mass_at_prediction. Population 100 cases / 100 decisions.

Exact Cloudflare/clef; adapter clef; settings {"questions_per_call": "all-case-questions"}. Run Cloudflare__clef.

Published measured S1MB run; community execution/submission status is not assumed independent. Choice is target mass at the selected option; soft targets are not ordinary hard-label accuracy. All published diagnostics: {"target_mass_at_prediction":0.95,"uniform_accuracy_baseline":0.3333333333333329,"fixed_answer_accuracy_baseline":0.51,"baseline_adjusted_skill":0.8979591836734693,"baseline_adjusted_score":0.8979591836734693}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0}. Parameter-count conventions excluding lookup/embedding parameters are not model MoE active counts. No executed weight SHA is stated in this display record; exact named repository and adapter are retained.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-18 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/argument_quality / test · binary_brier. Population 100 cases / 100 decisions.

Exact Cloudflare/clef; adapter clef; settings {"questions_per_call": "all-case-questions"}. Run Cloudflare__clef.

Published measured S1MB run; community execution/submission status is not assumed independent. All published diagnostics: {"binary_brier":0.06781713757807664,"accuracy":0.6549717725199998,"positive_recall":0.7718776966908456,"specificity":0.4674082822616405,"precision":0.6992688996911764,"balanced_accuracy":0.6196429894762431,"f1":0.7337814857996746,"true_positive":47.55028517899999,"false_positive":20.449714820999993,"false_negative":14.053107927,"true_negative":17.946892072999997,"positive_prevalence":0.6160339310599999,"majority_accuracy_baseline":0.6160339310599999,"constant_brier_baseline":0.079087076552024,"brier_skill":0.14250038647634067,"baseline_adjusted_skill":0.23928597895248616,"baseline_adjusted_score":0.23928597895248616}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0}. Parameter-count conventions excluding lookup/embedding parameters are not model MoE active counts. No executed weight SHA is stated in this display record; exact named repository and adapter are retained.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-18 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/babi_nli / test · binary_brier. Population 100 cases / 100 decisions.

Exact Cloudflare/clef; adapter clef; settings {"questions_per_call": "all-case-questions"}. Run Cloudflare__clef.

Published measured S1MB run; community execution/submission status is not assumed independent. All published diagnostics: {"binary_brier":0.10670396380122471,"accuracy":0.85,"positive_recall":0.82,"specificity":0.88,"precision":0.8723404255319149,"balanced_accuracy":0.85,"f1":0.845360824742268,"true_positive":41,"false_positive":6,"false_negative":9,"true_negative":44,"positive_prevalence":0.5,"majority_accuracy_baseline":0.5,"constant_brier_baseline":0.25,"brier_skill":0.5731841447951012,"baseline_adjusted_skill":0.7,"baseline_adjusted_score":0.7}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0}. Parameter-count conventions excluding lookup/embedding parameters are not model MoE active counts. No executed weight SHA is stated in this display record; exact named repository and adapter are retained.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-18 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/banking77 / test · target_mass_at_prediction. Population 100 cases / 100 decisions.

Exact Cloudflare/clef; adapter clef; settings {"questions_per_call": "all-case-questions"}. Run Cloudflare__clef.

Published measured S1MB run; community execution/submission status is not assumed independent. Choice is target mass at the selected option; soft targets are not ordinary hard-label accuracy. All published diagnostics: {"target_mass_at_prediction":0.94,"uniform_accuracy_baseline":0.012987012987012965,"fixed_answer_accuracy_baseline":0.05,"baseline_adjusted_skill":0.9368421052631578,"baseline_adjusted_score":0.9368421052631578}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0}. Parameter-count conventions excluding lookup/embedding parameters are not model MoE active counts. No executed weight SHA is stated in this display record; exact named repository and adapter are retained.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-18 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/bbq / test · target_mass_at_prediction. Population 100 cases / 100 decisions.

Exact Cloudflare/clef; adapter clef; settings {"questions_per_call": "all-case-questions"}. Run Cloudflare__clef.

Published measured S1MB run; community execution/submission status is not assumed independent. Choice is target mass at the selected option; soft targets are not ordinary hard-label accuracy. All published diagnostics: {"target_mass_at_prediction":0.89,"uniform_accuracy_baseline":0.3333333333333329,"fixed_answer_accuracy_baseline":0.4,"baseline_adjusted_skill":0.8166666666666667,"baseline_adjusted_score":0.8166666666666667}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0}. Parameter-count conventions excluding lookup/embedding parameters are not model MoE active counts. No executed weight SHA is stated in this display record; exact named repository and adapter are retained.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-18 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/boardgameqa / test · target_mass_at_prediction. Population 100 cases / 100 decisions.

Exact Cloudflare/clef; adapter clef; settings {"questions_per_call": "all-case-questions"}. Run Cloudflare__clef.

Published measured S1MB run; community execution/submission status is not assumed independent. Choice is target mass at the selected option; soft targets are not ordinary hard-label accuracy. All published diagnostics: {"target_mass_at_prediction":0.65,"uniform_accuracy_baseline":0.3333333333333329,"fixed_answer_accuracy_baseline":0.36,"baseline_adjusted_skill":0.45312500000000006,"baseline_adjusted_score":0.45312500000000006}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0}. Parameter-count conventions excluding lookup/embedding parameters are not model MoE active counts. No executed weight SHA is stated in this display record; exact named repository and adapter are retained.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-18 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/canttalk / test · binary_brier. Population 100 cases / 100 decisions.

Exact Cloudflare/clef; adapter clef; settings {"questions_per_call": "all-case-questions"}. Run Cloudflare__clef.

Published measured S1MB run; community execution/submission status is not assumed independent. All published diagnostics: {"binary_brier":0.1540570630667901,"accuracy":0.78,"positive_recall":1,"specificity":0.56,"precision":0.6944444444444444,"balanced_accuracy":0.78,"f1":0.819672131147541,"true_positive":50,"false_positive":22,"false_negative":0,"true_negative":28,"positive_prevalence":0.5,"majority_accuracy_baseline":0.5,"constant_brier_baseline":0.25,"brier_skill":0.3837717477328396,"baseline_adjusted_skill":0.56,"baseline_adjusted_score":0.56}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0}. Parameter-count conventions excluding lookup/embedding parameters are not model MoE active counts. No executed weight SHA is stated in this display record; exact named repository and adapter are retained.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-18 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/civil_comments / test · binary_brier. Population 100 cases / 700 decisions.

Exact Cloudflare/clef; adapter clef; settings {"questions_per_call": "all-case-questions"}. Run Cloudflare__clef.

Published measured S1MB run; community execution/submission status is not assumed independent. All published diagnostics: {"binary_brier":0.09230054258067365,"accuracy":0.6924679948861844,"positive_recall":0.5668360663938117,"specificity":0.7509668329818995,"precision":0.5145295361741161,"balanced_accuracy":0.6589014496878556,"f1":0.5394177466636177,"true_positive":126.05973636265844,"false_positive":118.94026363734156,"false_negative":96.33213994232938,"true_negative":358.6678600576706,"positive_prevalence":0.3177026804356969,"majority_accuracy_baseline":0.6822973195643032,"constant_brier_baseline":0.08872692397768597,"brier_skill":-0.04027659748337964,"baseline_adjusted_skill":0.31780289937571116,"baseline_adjusted_score":0.31780289937571116}. Counts: {"cases": 100, "expected": 700, "succeeded": 700, "failed": 0}. Parameter-count conventions excluding lookup/embedding parameters are not model MoE active counts. No executed weight SHA is stated in this display record; exact named repository and adapter are retained.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-18 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/cladder / test · binary_brier. Population 100 cases / 100 decisions.

Exact Cloudflare/clef; adapter clef; settings {"questions_per_call": "all-case-questions"}. Run Cloudflare__clef.

Published measured S1MB run; community execution/submission status is not assumed independent. All published diagnostics: {"binary_brier":0.042401443887513525,"accuracy":0.95,"positive_recall":0.94,"specificity":0.96,"precision":0.9591836734693877,"balanced_accuracy":0.95,"f1":0.9494949494949495,"true_positive":47,"false_positive":2,"false_negative":3,"true_negative":48,"positive_prevalence":0.5,"majority_accuracy_baseline":0.5,"constant_brier_baseline":0.25,"brier_skill":0.8303942244499459,"baseline_adjusted_skill":0.8999999999999999,"baseline_adjusted_score":0.8999999999999999}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0}. Parameter-count conventions excluding lookup/embedding parameters are not model MoE active counts. No executed weight SHA is stated in this display record; exact named repository and adapter are retained.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-18 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/clinc / test · target_mass_at_prediction. Population 151 cases / 151 decisions.

Exact Cloudflare/clef; adapter clef; settings {"questions_per_call": "all-case-questions"}. Run Cloudflare__clef.

Published measured S1MB run; community execution/submission status is not assumed independent. Choice is target mass at the selected option; soft targets are not ordinary hard-label accuracy. All published diagnostics: {"target_mass_at_prediction":0.9867549668874173,"uniform_accuracy_baseline":0.006622516556291403,"fixed_answer_accuracy_baseline":0.039735099337748346,"baseline_adjusted_skill":0.9862068965517242,"baseline_adjusted_score":0.9862068965517242}. Counts: {"cases": 151, "expected": 151, "succeeded": 151, "failed": 0}. Parameter-count conventions excluding lookup/embedding parameters are not model MoE active counts. No executed weight SHA is stated in this display record; exact named repository and adapter are retained.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-18 · Samples: 151

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/commonsensemoralja-noul / test · binary_brier. Population 300 cases / 300 decisions.

Exact Cloudflare/clef; adapter clef; settings {"questions_per_call": "all-case-questions"}. Run Cloudflare__clef.

Published measured S1MB run; community execution/submission status is not assumed independent. All published diagnostics: {"binary_brier":0.0803481965481978,"accuracy":0.9033333333333333,"positive_recall":0.8266666666666667,"specificity":0.98,"precision":0.9763779527559056,"balanced_accuracy":0.9033333333333333,"f1":0.8953068592057761,"true_positive":124,"false_positive":3,"false_negative":26,"true_negative":147,"positive_prevalence":0.5,"majority_accuracy_baseline":0.5,"constant_brier_baseline":0.25,"brier_skill":0.6786072138072088,"baseline_adjusted_skill":0.8066666666666666,"baseline_adjusted_score":0.8066666666666666}. Counts: {"cases": 300, "expected": 300, "succeeded": 300, "failed": 0}. Parameter-count conventions excluding lookup/embedding parameters are not model MoE active counts. No executed weight SHA is stated in this display record; exact named repository and adapter are retained.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-18 · Samples: 300

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/contract_nli / test · target_mass_at_prediction. Population 100 cases / 1700 decisions.

Exact Cloudflare/clef; adapter clef; settings {"questions_per_call": "all-case-questions"}. Run Cloudflare__clef.

Published measured S1MB run; community execution/submission status is not assumed independent. Choice is target mass at the selected option; soft targets are not ordinary hard-label accuracy. All published diagnostics: {"target_mass_at_prediction":0.8694117647058823,"uniform_accuracy_baseline":0.3333333333333301,"fixed_answer_accuracy_baseline":0.4576470588235294,"baseline_adjusted_skill":0.7592190889370932,"baseline_adjusted_score":0.7592190889370932}. Counts: {"cases": 100, "expected": 1700, "succeeded": 1700, "failed": 0}. Parameter-count conventions excluding lookup/embedding parameters are not model MoE active counts. No executed weight SHA is stated in this display record; exact named repository and adapter are retained.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-18 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/corr2cause / test · target_mass_at_prediction. Population 100 cases / 100 decisions.

Exact Cloudflare/clef; adapter clef; settings {"questions_per_call": "all-case-questions"}. Run Cloudflare__clef.

Published measured S1MB run; community execution/submission status is not assumed independent. Choice is target mass at the selected option; soft targets are not ordinary hard-label accuracy. All published diagnostics: {"target_mass_at_prediction":0.44,"uniform_accuracy_baseline":0.3333333333333329,"fixed_answer_accuracy_baseline":0.4,"baseline_adjusted_skill":0.06666666666666664,"baseline_adjusted_score":0.06666666666666664}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0}. Parameter-count conventions excluding lookup/embedding parameters are not model MoE active counts. No executed weight SHA is stated in this display record; exact named repository and adapter are retained.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-18 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/creak / test · binary_brier. Population 100 cases / 100 decisions.

Exact Cloudflare/clef; adapter clef; settings {"questions_per_call": "all-case-questions"}. Run Cloudflare__clef.

Published measured S1MB run; community execution/submission status is not assumed independent. All published diagnostics: {"binary_brier":0.09503811114948245,"accuracy":0.86,"positive_recall":0.98,"specificity":0.74,"precision":0.7903225806451613,"balanced_accuracy":0.86,"f1":0.875,"true_positive":49,"false_positive":13,"false_negative":1,"true_negative":37,"positive_prevalence":0.5,"majority_accuracy_baseline":0.5,"constant_brier_baseline":0.25,"brier_skill":0.6198475554020701,"baseline_adjusted_skill":0.72,"baseline_adjusted_score":0.72}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0}. Parameter-count conventions excluding lookup/embedding parameters are not model MoE active counts. No executed weight SHA is stated in this display record; exact named repository and adapter are retained.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-18 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/crows_pairs / test · target_mass_at_prediction. Population 100 cases / 100 decisions.

Exact Cloudflare/clef; adapter clef; settings {"questions_per_call": "all-case-questions"}. Run Cloudflare__clef.

Published measured S1MB run; community execution/submission status is not assumed independent. Choice is target mass at the selected option; soft targets are not ordinary hard-label accuracy. All published diagnostics: {"target_mass_at_prediction":0.81,"uniform_accuracy_baseline":0.5,"fixed_answer_accuracy_baseline":0.52,"baseline_adjusted_skill":0.6041666666666667,"baseline_adjusted_score":0.6041666666666667}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0}. Parameter-count conventions excluding lookup/embedding parameters are not model MoE active counts. No executed weight SHA is stated in this display record; exact named repository and adapter are retained.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-18 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/dbpedia / test · target_mass_at_prediction. Population 100 cases / 100 decisions.

Exact Cloudflare/clef; adapter clef; settings {"questions_per_call": "all-case-questions"}. Run Cloudflare__clef.

Published measured S1MB run; community execution/submission status is not assumed independent. Choice is target mass at the selected option; soft targets are not ordinary hard-label accuracy. All published diagnostics: {"target_mass_at_prediction":0.99,"uniform_accuracy_baseline":0.07142857142857136,"fixed_answer_accuracy_baseline":0.11,"baseline_adjusted_skill":0.9887640449438202,"baseline_adjusted_score":0.9887640449438202}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0}. Parameter-count conventions excluding lookup/embedding parameters are not model MoE active counts. No executed weight SHA is stated in this display record; exact named repository and adapter are retained.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-18 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/defeasible_nli / test · target_mass_at_prediction. Population 100 cases / 100 decisions.

Exact Cloudflare/clef; adapter clef; settings {"questions_per_call": "all-case-questions"}. Run Cloudflare__clef.

Published measured S1MB run; community execution/submission status is not assumed independent. Choice is target mass at the selected option; soft targets are not ordinary hard-label accuracy. All published diagnostics: {"target_mass_at_prediction":0.85,"uniform_accuracy_baseline":0.5,"fixed_answer_accuracy_baseline":0.54,"baseline_adjusted_skill":0.6739130434782608,"baseline_adjusted_score":0.6739130434782608}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0}. Parameter-count conventions excluding lookup/embedding parameters are not model MoE active counts. No executed weight SHA is stated in this display record; exact named repository and adapter are retained.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-18 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/esci / test · target_mass_at_prediction. Population 100 cases / 100 decisions.

Exact Cloudflare/clef; adapter clef; settings {"questions_per_call": "all-case-questions"}. Run Cloudflare__clef.

Published measured S1MB run; community execution/submission status is not assumed independent. Choice is target mass at the selected option; soft targets are not ordinary hard-label accuracy. All published diagnostics: {"target_mass_at_prediction":0.6,"uniform_accuracy_baseline":0.25,"fixed_answer_accuracy_baseline":0.27,"baseline_adjusted_skill":0.4520547945205479,"baseline_adjusted_score":0.4520547945205479}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0}. Parameter-count conventions excluding lookup/embedding parameters are not model MoE active counts. No executed weight SHA is stated in this display record; exact named repository and adapter are retained.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-18 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/ethics / test · target_mass_at_prediction. Population 18 cases / 18 decisions.

Exact Cloudflare/clef; adapter clef; settings {"questions_per_call": "all-case-questions"}. Run Cloudflare__clef.

Published measured S1MB run; community execution/submission status is not assumed independent. Choice is target mass at the selected option; soft targets are not ordinary hard-label accuracy. All published diagnostics: {"target_mass_at_prediction":0.8888888888888888,"uniform_accuracy_baseline":0.5,"fixed_answer_accuracy_baseline":0.6666666666666666,"baseline_adjusted_skill":0.6666666666666665,"baseline_adjusted_score":0.6666666666666665}. Counts: {"cases": 18, "expected": 18, "succeeded": 18, "failed": 0}. Parameter-count conventions excluding lookup/embedding parameters are not model MoE active counts. No executed weight SHA is stated in this display record; exact named repository and adapter are retained.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-18 · Samples: 18

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/ethics / test · binary_brier. Population 82 cases / 82 decisions.

Exact Cloudflare/clef; adapter clef; settings {"questions_per_call": "all-case-questions"}. Run Cloudflare__clef.

Published measured S1MB run; community execution/submission status is not assumed independent. All published diagnostics: {"binary_brier":0.13149566861908837,"accuracy":0.8414634146341463,"positive_recall":0.8,"specificity":0.8809523809523809,"precision":0.8648648648648649,"balanced_accuracy":0.8404761904761905,"f1":0.8311688311688312,"true_positive":32,"false_positive":5,"false_negative":8,"true_negative":37,"positive_prevalence":0.4878048780487805,"majority_accuracy_baseline":0.5121951219512195,"constant_brier_baseline":0.24985127900059498,"brier_skill":0.47370424059836314,"baseline_adjusted_skill":0.680952380952381,"baseline_adjusted_score":0.680952380952381}. Counts: {"cases": 82, "expected": 82, "succeeded": 82, "failed": 0}. Parameter-count conventions excluding lookup/embedding parameters are not model MoE active counts. No executed weight SHA is stated in this display record; exact named repository and adapter are retained.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-18 · Samples: 82

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/ethos / test · binary_brier. Population 100 cases / 100 decisions.

Exact Cloudflare/clef; adapter clef; settings {"questions_per_call": "all-case-questions"}. Run Cloudflare__clef.

Published measured S1MB run; community execution/submission status is not assumed independent. All published diagnostics: {"binary_brier":0.20611538470408838,"accuracy":0.72,"positive_recall":0.52,"specificity":0.92,"precision":0.8666666666666667,"balanced_accuracy":0.72,"f1":0.65,"true_positive":26,"false_positive":4,"false_negative":24,"true_negative":46,"positive_prevalence":0.5,"majority_accuracy_baseline":0.5,"constant_brier_baseline":0.25,"brier_skill":0.17553846118364647,"baseline_adjusted_skill":0.43999999999999995,"baseline_adjusted_score":0.43999999999999995}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0}. Parameter-count conventions excluding lookup/embedding parameters are not model MoE active counts. No executed weight SHA is stated in this display record; exact named repository and adapter are retained.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-18 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/few_nerd / test · target_mass_at_prediction. Population 100 cases / 358 decisions.

Exact Cloudflare/clef; adapter clef; settings {"questions_per_call": "all-case-questions"}. Run Cloudflare__clef.

Published measured S1MB run; community execution/submission status is not assumed independent. Choice is target mass at the selected option; soft targets are not ordinary hard-label accuracy. All published diagnostics: {"target_mass_at_prediction":0.8212290502793296,"uniform_accuracy_baseline":0.125,"fixed_answer_accuracy_baseline":0.2206703910614525,"baseline_adjusted_skill":0.7706093189964157,"baseline_adjusted_score":0.7706093189964157}. Counts: {"cases": 100, "expected": 358, "succeeded": 358, "failed": 0}. Parameter-count conventions excluding lookup/embedding parameters are not model MoE active counts. No executed weight SHA is stated in this display record; exact named repository and adapter are retained.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-18 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/fol_nli / test · target_mass_at_prediction. Population 100 cases / 100 decisions.

Exact Cloudflare/clef; adapter clef; settings {"questions_per_call": "all-case-questions"}. Run Cloudflare__clef.

Published measured S1MB run; community execution/submission status is not assumed independent. Choice is target mass at the selected option; soft targets are not ordinary hard-label accuracy. All published diagnostics: {"target_mass_at_prediction":0.59,"uniform_accuracy_baseline":0.3333333333333329,"fixed_answer_accuracy_baseline":0.39,"baseline_adjusted_skill":0.3278688524590163,"baseline_adjusted_score":0.3278688524590163}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0}. Parameter-count conventions excluding lookup/embedding parameters are not model MoE active counts. No executed weight SHA is stated in this display record; exact named repository and adapter are retained.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-18 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/followir_core17 / test · binary_brier. Population 100 cases / 200 decisions.

Exact Cloudflare/clef; adapter clef; settings {"questions_per_call": "all-case-questions"}. Run Cloudflare__clef.

Published measured S1MB run; community execution/submission status is not assumed independent. All published diagnostics: {"binary_brier":0.1546698639356976,"accuracy":0.815,"positive_recall":0.45454545454545453,"specificity":0.9166666666666666,"precision":0.6060606060606061,"balanced_accuracy":0.6856060606060606,"f1":0.5194805194805194,"true_positive":20,"false_positive":13,"false_negative":24,"true_negative":143,"positive_prevalence":0.22,"majority_accuracy_baseline":0.78,"constant_brier_baseline":0.17160000000000053,"brier_skill":0.09866046657519167,"baseline_adjusted_skill":0.3712121212121211,"baseline_adjusted_score":0.3712121212121211}. Counts: {"cases": 100, "expected": 200, "succeeded": 200, "failed": 0}. Parameter-count conventions excluding lookup/embedding parameters are not model MoE active counts. No executed weight SHA is stated in this display record; exact named repository and adapter are retained.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-18 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/followir_news21 / test · binary_brier. Population 100 cases / 200 decisions.

Exact Cloudflare/clef; adapter clef; settings {"questions_per_call": "all-case-questions"}. Run Cloudflare__clef.

Published measured S1MB run; community execution/submission status is not assumed independent. All published diagnostics: {"binary_brier":0.150535734157395,"accuracy":0.77,"positive_recall":0.36,"specificity":0.9066666666666666,"precision":0.5625,"balanced_accuracy":0.6333333333333333,"f1":0.43902439024390244,"true_positive":18,"false_positive":14,"false_negative":32,"true_negative":136,"positive_prevalence":0.25,"majority_accuracy_baseline":0.75,"constant_brier_baseline":0.1875,"brier_skill":0.19714275116056001,"baseline_adjusted_skill":0.2666666666666666,"baseline_adjusted_score":0.2666666666666666}. Counts: {"cases": 100, "expected": 200, "succeeded": 200, "failed": 0}. Parameter-count conventions excluding lookup/embedding parameters are not model MoE active counts. No executed weight SHA is stated in this display record; exact named repository and adapter are retained.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-18 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/followir_robust04 / test · binary_brier. Population 100 cases / 200 decisions.

Exact Cloudflare/clef; adapter clef; settings {"questions_per_call": "all-case-questions"}. Run Cloudflare__clef.

Published measured S1MB run; community execution/submission status is not assumed independent. All published diagnostics: {"binary_brier":0.04545814544176128,"accuracy":0.94,"positive_recall":0.7368421052631579,"specificity":0.9613259668508287,"precision":0.6666666666666666,"balanced_accuracy":0.8490840360569933,"f1":0.7,"true_positive":14,"false_positive":7,"false_negative":5,"true_negative":174,"positive_prevalence":0.095,"majority_accuracy_baseline":0.905,"constant_brier_baseline":0.0859749999999999,"brier_skill":0.47126321091292434,"baseline_adjusted_skill":0.6981680721139867,"baseline_adjusted_score":0.6981680721139867}. Counts: {"cases": 100, "expected": 200, "succeeded": 200, "failed": 0}. Parameter-count conventions excluding lookup/embedding parameters are not model MoE active counts. No executed weight SHA is stated in this display record; exact named repository and adapter are retained.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-18 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/generalization-japanese-diverse-choice / test · target_mass_at_prediction. Population 300 cases / 300 decisions.

Exact Cloudflare/clef; adapter clef; settings {"questions_per_call": "all-case-questions"}. Run Cloudflare__clef.

Published measured S1MB run; community execution/submission status is not assumed independent. Choice is target mass at the selected option; soft targets are not ordinary hard-label accuracy. All published diagnostics: {"target_mass_at_prediction":0.9966666666666667,"uniform_accuracy_baseline":0.3333333333333322,"fixed_answer_accuracy_baseline":0.3333333333333333,"baseline_adjusted_skill":0.9949999999999999,"baseline_adjusted_score":0.9949999999999999}. Counts: {"cases": 300, "expected": 300, "succeeded": 300, "failed": 0}. Parameter-count conventions excluding lookup/embedding parameters are not model MoE active counts. No executed weight SHA is stated in this display record; exact named repository and adapter are retained.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-18 · Samples: 300

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/generalization-japanese-diverse-noul / test · binary_brier. Population 300 cases / 300 decisions.

Exact Cloudflare/clef; adapter clef; settings {"questions_per_call": "all-case-questions"}. Run Cloudflare__clef.

Published measured S1MB run; community execution/submission status is not assumed independent. All published diagnostics: {"binary_brier":0.004024140687034135,"accuracy":1,"positive_recall":1,"specificity":1,"precision":1,"balanced_accuracy":1,"f1":1,"true_positive":150,"false_positive":0,"false_negative":0,"true_negative":150,"positive_prevalence":0.5,"majority_accuracy_baseline":0.5,"constant_brier_baseline":0.25,"brier_skill":0.9839034372518635,"baseline_adjusted_skill":1,"baseline_adjusted_score":1}. Counts: {"cases": 300, "expected": 300, "succeeded": 300, "failed": 0}. Parameter-count conventions excluding lookup/embedding parameters are not model MoE active counts. No executed weight SHA is stated in this display record; exact named repository and adapter are retained.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-18 · Samples: 300

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/generalization-japanese-knowledge-hard-choice / test · target_mass_at_prediction. Population 300 cases / 300 decisions.

Exact Cloudflare/clef; adapter clef; settings {"questions_per_call": "all-case-questions"}. Run Cloudflare__clef.

Published measured S1MB run; community execution/submission status is not assumed independent. Choice is target mass at the selected option; soft targets are not ordinary hard-label accuracy. All published diagnostics: {"target_mass_at_prediction":0.9933333333333333,"uniform_accuracy_baseline":0.25,"fixed_answer_accuracy_baseline":0.25,"baseline_adjusted_skill":0.991111111111111,"baseline_adjusted_score":0.991111111111111}. Counts: {"cases": 300, "expected": 300, "succeeded": 300, "failed": 0}. Parameter-count conventions excluding lookup/embedding parameters are not model MoE active counts. No executed weight SHA is stated in this display record; exact named repository and adapter are retained.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-18 · Samples: 300

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/generalization-japanese-knowledge-hard-noul / test · binary_brier. Population 300 cases / 300 decisions.

Exact Cloudflare/clef; adapter clef; settings {"questions_per_call": "all-case-questions"}. Run Cloudflare__clef.

Published measured S1MB run; community execution/submission status is not assumed independent. All published diagnostics: {"binary_brier":0.13022007256085394,"accuracy":0.7933333333333333,"positive_recall":0.9933333333333333,"specificity":0.5933333333333334,"precision":0.7095238095238096,"balanced_accuracy":0.7933333333333333,"f1":0.8277777777777777,"true_positive":149,"false_positive":61,"false_negative":1,"true_negative":89,"positive_prevalence":0.5,"majority_accuracy_baseline":0.5,"constant_brier_baseline":0.25,"brier_skill":0.47911970975658424,"baseline_adjusted_skill":0.5866666666666667,"baseline_adjusted_score":0.5866666666666667}. Counts: {"cases": 300, "expected": 300, "succeeded": 300, "failed": 0}. Parameter-count conventions excluding lookup/embedding parameters are not model MoE active counts. No executed weight SHA is stated in this display record; exact named repository and adapter are retained.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-18 · Samples: 300

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/generalization-japanese-knowledge-hard-score / test · normalized_expected_score_mae. Population 300 cases / 300 decisions.

Exact Cloudflare/clef; adapter clef; settings {"questions_per_call": "all-case-questions"}. Run Cloudflare__clef.

Published measured S1MB run; community execution/submission status is not assumed independent. All published diagnostics: {"normalized_expected_score_mae":0.14012516514087717,"normalized_score_rmse":0.1813443070484841,"constant_score_baseline":0.5,"constant_mae_baseline":0.3333333333333333,"baseline_adjusted_skill":0.5796245045773685,"baseline_adjusted_score":0.5796245045773685}. Counts: {"cases": 300, "expected": 300, "succeeded": 300, "failed": 0}. Parameter-count conventions excluding lookup/embedding parameters are not model MoE active counts. No executed weight SHA is stated in this display record; exact named repository and adapter are retained.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-18 · Samples: 300

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/generalization-japanese-math-formal-hard-choice / test · target_mass_at_prediction. Population 300 cases / 300 decisions.

Exact Cloudflare/clef; adapter clef; settings {"questions_per_call": "all-case-questions"}. Run Cloudflare__clef.

Published measured S1MB run; community execution/submission status is not assumed independent. Choice is target mass at the selected option; soft targets are not ordinary hard-label accuracy. All published diagnostics: {"target_mass_at_prediction":0.4633333333333333,"uniform_accuracy_baseline":0.25,"fixed_answer_accuracy_baseline":0.25,"baseline_adjusted_skill":0.28444444444444444,"baseline_adjusted_score":0.28444444444444444}. Counts: {"cases": 300, "expected": 300, "succeeded": 300, "failed": 0}. Parameter-count conventions excluding lookup/embedding parameters are not model MoE active counts. No executed weight SHA is stated in this display record; exact named repository and adapter are retained.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-18 · Samples: 300

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/generalization-japanese-math-formal-hard-noul / test · binary_brier. Population 300 cases / 300 decisions.

Exact Cloudflare/clef; adapter clef; settings {"questions_per_call": "all-case-questions"}. Run Cloudflare__clef.

Published measured S1MB run; community execution/submission status is not assumed independent. All published diagnostics: {"binary_brier":0.2756691284907727,"accuracy":0.5333333333333333,"positive_recall":0.26666666666666666,"specificity":0.8,"precision":0.5714285714285714,"balanced_accuracy":0.5333333333333333,"f1":0.36363636363636365,"true_positive":40,"false_positive":30,"false_negative":110,"true_negative":120,"positive_prevalence":0.5,"majority_accuracy_baseline":0.5,"constant_brier_baseline":0.25,"brier_skill":-0.10267651396309074,"baseline_adjusted_skill":0.06666666666666665,"baseline_adjusted_score":0.06666666666666665}. Counts: {"cases": 300, "expected": 300, "succeeded": 300, "failed": 0}. Parameter-count conventions excluding lookup/embedding parameters are not model MoE active counts. No executed weight SHA is stated in this display record; exact named repository and adapter are retained.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-18 · Samples: 300

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/generalization-japanese-math-formal-hard-score / test · normalized_expected_score_mae. Population 300 cases / 300 decisions.

Exact Cloudflare/clef; adapter clef; settings {"questions_per_call": "all-case-questions"}. Run Cloudflare__clef.

Published measured S1MB run; community execution/submission status is not assumed independent. All published diagnostics: {"normalized_expected_score_mae":0.2982529764048134,"normalized_score_rmse":0.3483959595329873,"constant_score_baseline":0.5,"constant_mae_baseline":0.3,"baseline_adjusted_skill":0.005823411983955307,"baseline_adjusted_score":0.005823411983955307}. Counts: {"cases": 300, "expected": 300, "succeeded": 300, "failed": 0}. Parameter-count conventions excluding lookup/embedding parameters are not model MoE active counts. No executed weight SHA is stated in this display record; exact named repository and adapter are retained.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-18 · Samples: 300

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/go_emotions / test · binary_brier. Population 100 cases / 2800 decisions.

Exact Cloudflare/clef; adapter clef; settings {"questions_per_call": "all-case-questions"}. Run Cloudflare__clef.

Published measured S1MB run; community execution/submission status is not assumed independent. All published diagnostics: {"binary_brier":0.042077939187331374,"accuracy":0.9081130952380956,"positive_recall":0.6211685348675727,"specificity":0.926430638491885,"precision":0.35022371364653243,"balanced_accuracy":0.7737995866797289,"f1":0.44790958835520905,"true_positive":104.36666666666673,"false_positive":193.63333333333344,"false_negative":63.65000000000001,"true_negative":2438.350000000001,"positive_prevalence":0.060005952380952354,"majority_accuracy_baseline":0.9399940476190476,"constant_brier_baseline":0.03220097218679296,"brier_skill":-0.3067288448076546,"baseline_adjusted_skill":0.5475991733594578,"baseline_adjusted_score":0.5475991733594578}. Counts: {"cases": 100, "expected": 2800, "succeeded": 2800, "failed": 0}. Parameter-count conventions excluding lookup/embedding parameters are not model MoE active counts. No executed weight SHA is stated in this display record; exact named repository and adapter are retained.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-18 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/gpqa-ja-main-choice / test · target_mass_at_prediction. Population 300 cases / 300 decisions.

Exact Cloudflare/clef; adapter clef; settings {"questions_per_call": "all-case-questions"}. Run Cloudflare__clef.

Published measured S1MB run; community execution/submission status is not assumed independent. Choice is target mass at the selected option; soft targets are not ordinary hard-label accuracy. All published diagnostics: {"target_mass_at_prediction":0.43333333333333335,"uniform_accuracy_baseline":0.25,"fixed_answer_accuracy_baseline":0.25,"baseline_adjusted_skill":0.24444444444444446,"baseline_adjusted_score":0.24444444444444446}. Counts: {"cases": 300, "expected": 300, "succeeded": 300, "failed": 0}. Parameter-count conventions excluding lookup/embedding parameters are not model MoE active counts. No executed weight SHA is stated in this display record; exact named repository and adapter are retained.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-18 · Samples: 300

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/gretel_pii / test · target_mass_at_prediction. Population 100 cases / 353 decisions.

Exact Cloudflare/clef; adapter clef; settings {"questions_per_call": "all-case-questions"}. Run Cloudflare__clef.

Published measured S1MB run; community execution/submission status is not assumed independent. Choice is target mass at the selected option; soft targets are not ordinary hard-label accuracy. All published diagnostics: {"target_mass_at_prediction":0.7903682719546742,"uniform_accuracy_baseline":0.027777777777777853,"fixed_answer_accuracy_baseline":0.1274787535410765,"baseline_adjusted_skill":0.7597402597402597,"baseline_adjusted_score":0.7597402597402597}. Counts: {"cases": 100, "expected": 353, "succeeded": 353, "failed": 0}. Parameter-count conventions excluding lookup/embedding parameters are not model MoE active counts. No executed weight SHA is stated in this display record; exact named repository and adapter are retained.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-18 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/gsm8k / test · target_mass_at_prediction. Population 100 cases / 100 decisions.

Exact Cloudflare/clef; adapter clef; settings {"questions_per_call": "all-case-questions"}. Run Cloudflare__clef.

Published measured S1MB run; community execution/submission status is not assumed independent. Choice is target mass at the selected option; soft targets are not ordinary hard-label accuracy. All published diagnostics: {"target_mass_at_prediction":0.83,"uniform_accuracy_baseline":0.25,"fixed_answer_accuracy_baseline":0.3,"baseline_adjusted_skill":0.7571428571428572,"baseline_adjusted_score":0.7571428571428572}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0}. Parameter-count conventions excluding lookup/embedding parameters are not model MoE active counts. No executed weight SHA is stated in this display record; exact named repository and adapter are retained.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-18 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/hans / test · binary_brier. Population 100 cases / 100 decisions.

Exact Cloudflare/clef; adapter clef; settings {"questions_per_call": "all-case-questions"}. Run Cloudflare__clef.

Published measured S1MB run; community execution/submission status is not assumed independent. All published diagnostics: {"binary_brier":0.0673819323363518,"accuracy":0.94,"positive_recall":1,"specificity":0.88,"precision":0.8928571428571429,"balanced_accuracy":0.94,"f1":0.9433962264150944,"true_positive":50,"false_positive":6,"false_negative":0,"true_negative":44,"positive_prevalence":0.5,"majority_accuracy_baseline":0.5,"constant_brier_baseline":0.25,"brier_skill":0.7304722706545927,"baseline_adjusted_skill":0.8799999999999999,"baseline_adjusted_score":0.8799999999999999}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0}. Parameter-count conventions excluding lookup/embedding parameters are not model MoE active counts. No executed weight SHA is stated in this display record; exact named repository and adapter are retained.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-18 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/hatecheck / test · binary_brier. Population 100 cases / 100 decisions.

Exact Cloudflare/clef; adapter clef; settings {"questions_per_call": "all-case-questions"}. Run Cloudflare__clef.

Published measured S1MB run; community execution/submission status is not assumed independent. All published diagnostics: {"binary_brier":0.11287172771733159,"accuracy":0.83,"positive_recall":0.94,"specificity":0.72,"precision":0.7704918032786885,"balanced_accuracy":0.83,"f1":0.8468468468468469,"true_positive":47,"false_positive":14,"false_negative":3,"true_negative":36,"positive_prevalence":0.5,"majority_accuracy_baseline":0.5,"constant_brier_baseline":0.25,"brier_skill":0.5485130891306736,"baseline_adjusted_skill":0.6599999999999999,"baseline_adjusted_score":0.6599999999999999}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0}. Parameter-count conventions excluding lookup/embedding parameters are not model MoE active counts. No executed weight SHA is stated in this display record; exact named repository and adapter are retained.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-18 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/hh_rlhf / test · target_mass_at_prediction. Population 100 cases / 100 decisions.

Exact Cloudflare/clef; adapter clef; settings {"questions_per_call": "all-case-questions"}. Run Cloudflare__clef.

Published measured S1MB run; community execution/submission status is not assumed independent. Choice is target mass at the selected option; soft targets are not ordinary hard-label accuracy. All published diagnostics: {"target_mass_at_prediction":0.63,"uniform_accuracy_baseline":0.5,"fixed_answer_accuracy_baseline":0.61,"baseline_adjusted_skill":0.05128205128205133,"baseline_adjusted_score":0.05128205128205133}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0}. Parameter-count conventions excluding lookup/embedding parameters are not model MoE active counts. No executed weight SHA is stated in this display record; exact named repository and adapter are retained.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-18 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/hwu64 / test · target_mass_at_prediction. Population 100 cases / 100 decisions.

Exact Cloudflare/clef; adapter clef; settings {"questions_per_call": "all-case-questions"}. Run Cloudflare__clef.

Published measured S1MB run; community execution/submission status is not assumed independent. Choice is target mass at the selected option; soft targets are not ordinary hard-label accuracy. All published diagnostics: {"target_mass_at_prediction":0.92,"uniform_accuracy_baseline":0.015625,"fixed_answer_accuracy_baseline":0.06,"baseline_adjusted_skill":0.9148936170212768,"baseline_adjusted_score":0.9148936170212768}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0}. Parameter-count conventions excluding lookup/embedding parameters are not model MoE active counts. No executed weight SHA is stated in this display record; exact named repository and adapter are retained.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-18 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/impli / test · binary_brier. Population 100 cases / 100 decisions.

Exact Cloudflare/clef; adapter clef; settings {"questions_per_call": "all-case-questions"}. Run Cloudflare__clef.

Published measured S1MB run; community execution/submission status is not assumed independent. All published diagnostics: {"binary_brier":0.11992722797148044,"accuracy":0.82,"positive_recall":0.76,"specificity":0.88,"precision":0.8636363636363636,"balanced_accuracy":0.8200000000000001,"f1":0.8085106382978723,"true_positive":38,"false_positive":6,"false_negative":12,"true_negative":44,"positive_prevalence":0.5,"majority_accuracy_baseline":0.5,"constant_brier_baseline":0.25,"brier_skill":0.5202910881140783,"baseline_adjusted_skill":0.6400000000000001,"baseline_adjusted_score":0.6400000000000001}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0}. Parameter-count conventions excluding lookup/embedding parameters are not model MoE active counts. No executed weight SHA is stated in this display record; exact named repository and adapter are retained.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-18 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/jamc-qa-choice / test · target_mass_at_prediction. Population 300 cases / 300 decisions.

Exact Cloudflare/clef; adapter clef; settings {"questions_per_call": "all-case-questions"}. Run Cloudflare__clef.

Published measured S1MB run; community execution/submission status is not assumed independent. Choice is target mass at the selected option; soft targets are not ordinary hard-label accuracy. All published diagnostics: {"target_mass_at_prediction":0.49666666666666665,"uniform_accuracy_baseline":0.25,"fixed_answer_accuracy_baseline":0.25,"baseline_adjusted_skill":0.32888888888888884,"baseline_adjusted_score":0.32888888888888884}. Counts: {"cases": 300, "expected": 300, "succeeded": 300, "failed": 0}. Parameter-count conventions excluding lookup/embedding parameters are not model MoE active counts. No executed weight SHA is stated in this display record; exact named repository and adapter are retained.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-18 · Samples: 300

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/jamp-choice / test · target_mass_at_prediction. Population 300 cases / 300 decisions.

Exact Cloudflare/clef; adapter clef; settings {"questions_per_call": "all-case-questions"}. Run Cloudflare__clef.

Published measured S1MB run; community execution/submission status is not assumed independent. Choice is target mass at the selected option; soft targets are not ordinary hard-label accuracy. All published diagnostics: {"target_mass_at_prediction":0.69,"uniform_accuracy_baseline":0.3333333333333322,"fixed_answer_accuracy_baseline":0.3333333333333333,"baseline_adjusted_skill":0.5349999999999999,"baseline_adjusted_score":0.5349999999999999}. Counts: {"cases": 300, "expected": 300, "succeeded": 300, "failed": 0}. Parameter-count conventions excluding lookup/embedding parameters are not model MoE active counts. No executed weight SHA is stated in this display record; exact named repository and adapter are retained.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-18 · Samples: 300

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/janli-choice / test · target_mass_at_prediction. Population 285 cases / 285 decisions.

Exact Cloudflare/clef; adapter clef; settings {"questions_per_call": "all-case-questions"}. Run Cloudflare__clef.

Published measured S1MB run; community execution/submission status is not assumed independent. Choice is target mass at the selected option; soft targets are not ordinary hard-label accuracy. All published diagnostics: {"target_mass_at_prediction":0.6771929824561403,"uniform_accuracy_baseline":0.5,"fixed_answer_accuracy_baseline":0.5263157894736842,"baseline_adjusted_skill":0.31851851851851853,"baseline_adjusted_score":0.31851851851851853}. Counts: {"cases": 285, "expected": 285, "succeeded": 285, "failed": 0}. Parameter-count conventions excluding lookup/embedding parameters are not model MoE active counts. No executed weight SHA is stated in this display record; exact named repository and adapter are retained.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-18 · Samples: 285

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/jcola-noul / test · binary_brier. Population 299 cases / 299 decisions.

Exact Cloudflare/clef; adapter clef; settings {"questions_per_call": "all-case-questions"}. Run Cloudflare__clef.

Published measured S1MB run; community execution/submission status is not assumed independent. All published diagnostics: {"binary_brier":0.19814350762880695,"accuracy":0.6722408026755853,"positive_recall":0.8066666666666666,"specificity":0.5369127516778524,"precision":0.6368421052631579,"balanced_accuracy":0.6717897091722596,"f1":0.711764705882353,"true_positive":121,"false_positive":69,"false_negative":29,"true_negative":80,"positive_prevalence":0.5016722408026756,"majority_accuracy_baseline":0.5016722408026756,"constant_brier_baseline":0.24999720361069733,"brier_skill":0.20741710400344482,"baseline_adjusted_skill":0.3435794183445191,"baseline_adjusted_score":0.3435794183445191}. Counts: {"cases": 299, "expected": 299, "succeeded": 299, "failed": 0}. Parameter-count conventions excluding lookup/embedding parameters are not model MoE active counts. No executed weight SHA is stated in this display record; exact named repository and adapter are retained.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-18 · Samples: 299

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/jcommonsenseqa-choice / test · target_mass_at_prediction. Population 300 cases / 300 decisions.

Exact Cloudflare/clef; adapter clef; settings {"questions_per_call": "all-case-questions"}. Run Cloudflare__clef.

Published measured S1MB run; community execution/submission status is not assumed independent. Choice is target mass at the selected option; soft targets are not ordinary hard-label accuracy. All published diagnostics: {"target_mass_at_prediction":0.9466666666666667,"uniform_accuracy_baseline":0.20000000000000104,"fixed_answer_accuracy_baseline":0.20000000000000104,"baseline_adjusted_skill":0.9333333333333332,"baseline_adjusted_score":0.9333333333333332}. Counts: {"cases": 300, "expected": 300, "succeeded": 300, "failed": 0}. Parameter-count conventions excluding lookup/embedding parameters are not model MoE active counts. No executed weight SHA is stated in this display record; exact named repository and adapter are retained.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-18 · Samples: 300

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/jculture-mcq-noul / test · binary_brier. Population 283 cases / 283 decisions.

Exact Cloudflare/clef; adapter clef; settings {"questions_per_call": "all-case-questions"}. Run Cloudflare__clef.

Published measured S1MB run; community execution/submission status is not assumed independent. All published diagnostics: {"binary_brier":0.26024002125135315,"accuracy":0.5618374558303887,"positive_recall":0.8045112781954887,"specificity":0.3466666666666667,"precision":0.5219512195121951,"balanced_accuracy":0.5755889724310777,"f1":0.6331360946745562,"true_positive":107,"false_positive":98,"false_negative":26,"true_negative":52,"positive_prevalence":0.46996466431095407,"majority_accuracy_baseline":0.5300353356890459,"constant_brier_baseline":0.2490978786100456,"brier_skill":-0.04472997804509693,"baseline_adjusted_skill":0.1511779448621553,"baseline_adjusted_score":0.1511779448621553}. Counts: {"cases": 283, "expected": 283, "succeeded": 283, "failed": 0}. Parameter-count conventions excluding lookup/embedding parameters are not model MoE active counts. No executed weight SHA is stated in this display record; exact named repository and adapter are retained.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-18 · Samples: 283

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/jmmlu-japan-choice / test · target_mass_at_prediction. Population 300 cases / 300 decisions.

Exact Cloudflare/clef; adapter clef; settings {"questions_per_call": "all-case-questions"}. Run Cloudflare__clef.

Published measured S1MB run; community execution/submission status is not assumed independent. Choice is target mass at the selected option; soft targets are not ordinary hard-label accuracy. All published diagnostics: {"target_mass_at_prediction":0.8533333333333334,"uniform_accuracy_baseline":0.25,"fixed_answer_accuracy_baseline":0.25,"baseline_adjusted_skill":0.8044444444444445,"baseline_adjusted_score":0.8044444444444445}. Counts: {"cases": 300, "expected": 300, "succeeded": 300, "failed": 0}. Parameter-count conventions excluding lookup/embedding parameters are not model MoE active counts. No executed weight SHA is stated in this display record; exact named repository and adapter are retained.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-18 · Samples: 300

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/jmteb-retrieval / test · target_mass_at_prediction. Population 300 cases / 300 decisions.

Exact Cloudflare/clef; adapter clef; settings {"questions_per_call": "all-case-questions"}. Run Cloudflare__clef.

Published measured S1MB run; community execution/submission status is not assumed independent. Choice is target mass at the selected option; soft targets are not ordinary hard-label accuracy. All published diagnostics: {"target_mass_at_prediction":0.9633333333333334,"uniform_accuracy_baseline":0.2061666666666668,"fixed_answer_accuracy_baseline":0.21333333333333335,"baseline_adjusted_skill":0.9533898305084746,"baseline_adjusted_score":0.9533898305084746}. Counts: {"cases": 300, "expected": 300, "succeeded": 300, "failed": 0}. Parameter-count conventions excluding lookup/embedding parameters are not model MoE active counts. No executed weight SHA is stated in this display record; exact named repository and adapter are retained.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-18 · Samples: 300

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/jnli-choice / test · target_mass_at_prediction. Population 300 cases / 300 decisions.

Exact Cloudflare/clef; adapter clef; settings {"questions_per_call": "all-case-questions"}. Run Cloudflare__clef.

Published measured S1MB run; community execution/submission status is not assumed independent. Choice is target mass at the selected option; soft targets are not ordinary hard-label accuracy. All published diagnostics: {"target_mass_at_prediction":0.8266666666666667,"uniform_accuracy_baseline":0.3333333333333322,"fixed_answer_accuracy_baseline":0.3333333333333333,"baseline_adjusted_skill":0.74,"baseline_adjusted_score":0.74}. Counts: {"cases": 300, "expected": 300, "succeeded": 300, "failed": 0}. Parameter-count conventions excluding lookup/embedding parameters are not model MoE active counts. No executed weight SHA is stated in this display record; exact named repository and adapter are retained.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-18 · Samples: 300

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/jsem-choice / test · target_mass_at_prediction. Population 300 cases / 300 decisions.

Exact Cloudflare/clef; adapter clef; settings {"questions_per_call": "all-case-questions"}. Run Cloudflare__clef.

Published measured S1MB run; community execution/submission status is not assumed independent. Choice is target mass at the selected option; soft targets are not ordinary hard-label accuracy. All published diagnostics: {"target_mass_at_prediction":0.6233333333333333,"uniform_accuracy_baseline":0.25,"fixed_answer_accuracy_baseline":0.37333333333333335,"baseline_adjusted_skill":0.39893617021276584,"baseline_adjusted_score":0.39893617021276584}. Counts: {"cases": 300, "expected": 300, "succeeded": 300, "failed": 0}. Parameter-count conventions excluding lookup/embedding parameters are not model MoE active counts. No executed weight SHA is stated in this display record; exact named repository and adapter are retained.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-18 · Samples: 300

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/jsick-sts-score / test · normalized_expected_score_mae. Population 300 cases / 300 decisions.

Exact Cloudflare/clef; adapter clef; settings {"questions_per_call": "all-case-questions"}. Run Cloudflare__clef.

Published measured S1MB run; community execution/submission status is not assumed independent. All published diagnostics: {"normalized_expected_score_mae":0.15973096203272397,"normalized_score_rmse":0.1969362565805731,"constant_score_baseline":0.5,"constant_mae_baseline":0.26208333333333317,"baseline_adjusted_skill":0.3905336901772055,"baseline_adjusted_score":0.3905336901772055}. Counts: {"cases": 300, "expected": 300, "succeeded": 300, "failed": 0}. Parameter-count conventions excluding lookup/embedding parameters are not model MoE active counts. No executed weight SHA is stated in this display record; exact named repository and adapter are retained.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-18 · Samples: 300

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/jsts-score / test · normalized_expected_score_mae. Population 300 cases / 300 decisions.

Exact Cloudflare/clef; adapter clef; settings {"questions_per_call": "all-case-questions"}. Run Cloudflare__clef.

Published measured S1MB run; community execution/submission status is not assumed independent. All published diagnostics: {"normalized_expected_score_mae":0.09103160013398158,"normalized_score_rmse":0.11594904608095581,"constant_score_baseline":0.52,"constant_mae_baseline":0.24733333333333307,"baseline_adjusted_skill":0.6319477083531738,"baseline_adjusted_score":0.6319477083531738}. Counts: {"cases": 300, "expected": 300, "succeeded": 300, "failed": 0}. Parameter-count conventions excluding lookup/embedding parameters are not model MoE active counts. No executed weight SHA is stated in this display record; exact named repository and adapter are retained.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-18 · Samples: 300

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/kuci-choice / test · target_mass_at_prediction. Population 300 cases / 300 decisions.

Exact Cloudflare/clef; adapter clef; settings {"questions_per_call": "all-case-questions"}. Run Cloudflare__clef.

Published measured S1MB run; community execution/submission status is not assumed independent. Choice is target mass at the selected option; soft targets are not ordinary hard-label accuracy. All published diagnostics: {"target_mass_at_prediction":0.78,"uniform_accuracy_baseline":0.25,"fixed_answer_accuracy_baseline":0.25,"baseline_adjusted_skill":0.7066666666666667,"baseline_adjusted_score":0.7066666666666667}. Counts: {"cases": 300, "expected": 300, "succeeded": 300, "failed": 0}. Parameter-count conventions excluding lookup/embedding parameters are not model MoE active counts. No executed weight SHA is stated in this display record; exact named repository and adapter are retained.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-18 · Samples: 300

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/laya__ag_news / test · target_mass_at_prediction. Population 100 cases / 100 decisions.

Exact Cloudflare/clef; adapter clef; settings {"questions_per_call": "all-case-questions"}. Run Cloudflare__clef.

Published measured S1MB run; community execution/submission status is not assumed independent. Choice is target mass at the selected option; soft targets are not ordinary hard-label accuracy. All published diagnostics: {"target_mass_at_prediction":0.92,"uniform_accuracy_baseline":0.25,"fixed_answer_accuracy_baseline":0.33,"baseline_adjusted_skill":0.8805970149253733,"baseline_adjusted_score":0.8805970149253733}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0}. Parameter-count conventions excluding lookup/embedding parameters are not model MoE active counts. No executed weight SHA is stated in this display record; exact named repository and adapter are retained.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-18 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/laya__enron_spam / test · binary_brier. Population 100 cases / 100 decisions.

Exact Cloudflare/clef; adapter clef; settings {"questions_per_call": "all-case-questions"}. Run Cloudflare__clef.

Published measured S1MB run; community execution/submission status is not assumed independent. All published diagnostics: {"binary_brier":0.031191096477811268,"accuracy":0.96,"positive_recall":0.96,"specificity":0.96,"precision":0.96,"balanced_accuracy":0.96,"f1":0.96,"true_positive":48,"false_positive":2,"false_negative":2,"true_negative":48,"positive_prevalence":0.5,"majority_accuracy_baseline":0.5,"constant_brier_baseline":0.25,"brier_skill":0.8752356140887549,"baseline_adjusted_skill":0.9199999999999999,"baseline_adjusted_score":0.9199999999999999}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0}. Parameter-count conventions excluding lookup/embedding parameters are not model MoE active counts. No executed weight SHA is stated in this display record; exact named repository and adapter are retained.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-18 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/laya__typed_decisions / test · target_mass_at_prediction. Population 100 cases / 148 decisions.

Exact Cloudflare/clef; adapter clef; settings {"questions_per_call": "all-case-questions"}. Run Cloudflare__clef.

Published measured S1MB run; community execution/submission status is not assumed independent. Choice is target mass at the selected option; soft targets are not ordinary hard-label accuracy. All published diagnostics: {"target_mass_at_prediction":0.48200570945945975,"uniform_accuracy_baseline":0.23445945945945923,"fixed_answer_accuracy_baseline":0.2623439054054054,"baseline_adjusted_skill":0.2977834869984737,"baseline_adjusted_score":0.2977834869984737}. Counts: {"cases": 100, "expected": 148, "succeeded": 148, "failed": 0}. Parameter-count conventions excluding lookup/embedding parameters are not model MoE active counts. No executed weight SHA is stated in this display record; exact named repository and adapter are retained.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-18 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/laya__typed_decisions / test · binary_brier. Population 100 cases / 152 decisions.

Exact Cloudflare/clef; adapter clef; settings {"questions_per_call": "all-case-questions"}. Run Cloudflare__clef.

Published measured S1MB run; community execution/submission status is not assumed independent. All published diagnostics: {"binary_brier":0.0641008045840347,"accuracy":0.7063596381578946,"positive_recall":0.900368644441517,"specificity":0.5216199918425376,"precision":0.6418589807692306,"balanced_accuracy":0.7109943181420273,"f1":0.7494479945641406,"true_positive":66.75333399999998,"false_positive":37.246666000000005,"false_negative":7.386669,"true_negative":40.613330999999995,"positive_prevalence":0.48776317763157917,"majority_accuracy_baseline":0.5122368223684208,"constant_brier_baseline":0.0945191003976856,"brier_skill":0.32182168139208944,"baseline_adjusted_skill":0.4219886362840546,"baseline_adjusted_score":0.4219886362840546}. Counts: {"cases": 100, "expected": 152, "succeeded": 152, "failed": 0}. Parameter-count conventions excluding lookup/embedding parameters are not model MoE active counts. No executed weight SHA is stated in this display record; exact named repository and adapter are retained.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-18 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/laya__typed_decisions / test · normalized_expected_score_mae. Population 100 cases / 200 decisions.

Exact Cloudflare/clef; adapter clef; settings {"questions_per_call": "all-case-questions"}. Run Cloudflare__clef.

Published measured S1MB run; community execution/submission status is not assumed independent. All published diagnostics: {"normalized_expected_score_mae":0.09499809716231024,"normalized_score_rmse":0.13075660703227174,"constant_score_baseline":0.5841668333333334,"constant_mae_baseline":0.21700552708333315,"baseline_adjusted_skill":0.5622319005458803,"baseline_adjusted_score":0.5622319005458803}. Counts: {"cases": 100, "expected": 200, "succeeded": 200, "failed": 0}. Parameter-count conventions excluding lookup/embedding parameters are not model MoE active counts. No executed weight SHA is stated in this display record; exact named repository and adapter are retained.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-18 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/lexcomp / test · binary_brier. Population 100 cases / 100 decisions.

Exact Cloudflare/clef; adapter clef; settings {"questions_per_call": "all-case-questions"}. Run Cloudflare__clef.

Published measured S1MB run; community execution/submission status is not assumed independent. All published diagnostics: {"binary_brier":0.19493887572279522,"accuracy":0.71,"positive_recall":0.74,"specificity":0.68,"precision":0.6981132075471698,"balanced_accuracy":0.71,"f1":0.7184466019417476,"true_positive":37,"false_positive":16,"false_negative":13,"true_negative":34,"positive_prevalence":0.5,"majority_accuracy_baseline":0.5,"constant_brier_baseline":0.25,"brier_skill":0.2202444971088191,"baseline_adjusted_skill":0.41999999999999993,"baseline_adjusted_score":0.41999999999999993}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0}. Parameter-count conventions excluding lookup/embedding parameters are not model MoE active counts. No executed weight SHA is stated in this display record; exact named repository and adapter are retained.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-18 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/logical_entailment / test · binary_brier. Population 100 cases / 100 decisions.

Exact Cloudflare/clef; adapter clef; settings {"questions_per_call": "all-case-questions"}. Run Cloudflare__clef.

Published measured S1MB run; community execution/submission status is not assumed independent. All published diagnostics: {"binary_brier":0.32030538399522385,"accuracy":0.57,"positive_recall":0.2,"specificity":0.94,"precision":0.7692307692307693,"balanced_accuracy":0.57,"f1":0.31746031746031744,"true_positive":10,"false_positive":3,"false_negative":40,"true_negative":47,"positive_prevalence":0.5,"majority_accuracy_baseline":0.5,"constant_brier_baseline":0.25,"brier_skill":-0.2812215359808954,"baseline_adjusted_skill":0.1399999999999999,"baseline_adjusted_score":0.1399999999999999}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0}. Parameter-count conventions excluding lookup/embedding parameters are not model MoE active counts. No executed weight SHA is stated in this display record; exact named repository and adapter are retained.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-18 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/lonli / test · target_mass_at_prediction. Population 100 cases / 100 decisions.

Exact Cloudflare/clef; adapter clef; settings {"questions_per_call": "all-case-questions"}. Run Cloudflare__clef.

Published measured S1MB run; community execution/submission status is not assumed independent. Choice is target mass at the selected option; soft targets are not ordinary hard-label accuracy. All published diagnostics: {"target_mass_at_prediction":0.86,"uniform_accuracy_baseline":0.3333333333333329,"fixed_answer_accuracy_baseline":0.39,"baseline_adjusted_skill":0.7704918032786885,"baseline_adjusted_score":0.7704918032786885}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0}. Parameter-count conventions excluding lookup/embedding parameters are not model MoE active counts. No executed weight SHA is stated in this display record; exact named repository and adapter are retained.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-18 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/massive / test · target_mass_at_prediction. Population 100 cases / 200 decisions.

Exact Cloudflare/clef; adapter clef; settings {"questions_per_call": "all-case-questions"}. Run Cloudflare__clef.

Published measured S1MB run; community execution/submission status is not assumed independent. Choice is target mass at the selected option; soft targets are not ordinary hard-label accuracy. All published diagnostics: {"target_mass_at_prediction":0.95,"uniform_accuracy_baseline":0.03611111111111102,"fixed_answer_accuracy_baseline":0.08833333333333329,"baseline_adjusted_skill":0.9451553930530164,"baseline_adjusted_score":0.9451553930530164}. Counts: {"cases": 100, "expected": 200, "succeeded": 200, "failed": 0}. Parameter-count conventions excluding lookup/embedding parameters are not model MoE active counts. No executed weight SHA is stated in this display record; exact named repository and adapter are retained.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-18 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/mintaka-ja-boolean-noul / test · binary_brier. Population 300 cases / 300 decisions.

Exact Cloudflare/clef; adapter clef; settings {"questions_per_call": "all-case-questions"}. Run Cloudflare__clef.

Published measured S1MB run; community execution/submission status is not assumed independent. All published diagnostics: {"binary_brier":0.14549292831388055,"accuracy":0.7766666666666666,"positive_recall":0.7933333333333333,"specificity":0.76,"precision":0.7677419354838709,"balanced_accuracy":0.7766666666666666,"f1":0.780327868852459,"true_positive":119,"false_positive":36,"false_negative":31,"true_negative":114,"positive_prevalence":0.5,"majority_accuracy_baseline":0.5,"constant_brier_baseline":0.25,"brier_skill":0.4180282867444778,"baseline_adjusted_skill":0.5533333333333332,"baseline_adjusted_score":0.5533333333333332}. Counts: {"cases": 300, "expected": 300, "succeeded": 300, "failed": 0}. Parameter-count conventions excluding lookup/embedding parameters are not model MoE active counts. No executed weight SHA is stated in this display record; exact named repository and adapter are retained.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-18 · Samples: 300

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/miqa / test · target_mass_at_prediction. Population 100 cases / 100 decisions.

Exact Cloudflare/clef; adapter clef; settings {"questions_per_call": "all-case-questions"}. Run Cloudflare__clef.

Published measured S1MB run; community execution/submission status is not assumed independent. Choice is target mass at the selected option; soft targets are not ordinary hard-label accuracy. All published diagnostics: {"target_mass_at_prediction":0.97,"uniform_accuracy_baseline":0.5,"fixed_answer_accuracy_baseline":0.56,"baseline_adjusted_skill":0.9318181818181818,"baseline_adjusted_score":0.9318181818181818}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0}. Parameter-count conventions excluding lookup/embedding parameters are not model MoE active counts. No executed weight SHA is stated in this display record; exact named repository and adapter are retained.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-18 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/mmlu-prox-ja-choice / test · target_mass_at_prediction. Population 300 cases / 300 decisions.

Exact Cloudflare/clef; adapter clef; settings {"questions_per_call": "all-case-questions"}. Run Cloudflare__clef.

Published measured S1MB run; community execution/submission status is not assumed independent. Choice is target mass at the selected option; soft targets are not ordinary hard-label accuracy. All published diagnostics: {"target_mass_at_prediction":0.6133333333333333,"uniform_accuracy_baseline":0.11021428571428617,"fixed_answer_accuracy_baseline":0.12554761904761905,"baseline_adjusted_skill":0.5578184986522177,"baseline_adjusted_score":0.5578184986522177}. Counts: {"cases": 300, "expected": 300, "succeeded": 300, "failed": 0}. Parameter-count conventions excluding lookup/embedding parameters are not model MoE active counts. No executed weight SHA is stated in this display record; exact named repository and adapter are retained.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-18 · Samples: 300

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/mmmlu-choice / test · target_mass_at_prediction. Population 300 cases / 300 decisions.

Exact Cloudflare/clef; adapter clef; settings {"questions_per_call": "all-case-questions"}. Run Cloudflare__clef.

Published measured S1MB run; community execution/submission status is not assumed independent. Choice is target mass at the selected option; soft targets are not ordinary hard-label accuracy. All published diagnostics: {"target_mass_at_prediction":0.8033333333333333,"uniform_accuracy_baseline":0.25,"fixed_answer_accuracy_baseline":0.25,"baseline_adjusted_skill":0.7377777777777778,"baseline_adjusted_score":0.7377777777777778}. Counts: {"cases": 300, "expected": 300, "succeeded": 300, "failed": 0}. Parameter-count conventions excluding lookup/embedding parameters are not model MoE active counts. No executed weight SHA is stated in this display record; exact named repository and adapter are retained.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-18 · Samples: 300

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/mtop / test · target_mass_at_prediction. Population 100 cases / 200 decisions.

Exact Cloudflare/clef; adapter clef; settings {"questions_per_call": "all-case-questions"}. Run Cloudflare__clef.

Published measured S1MB run; community execution/submission status is not assumed independent. Choice is target mass at the selected option; soft targets are not ordinary hard-label accuracy. All published diagnostics: {"target_mass_at_prediction":0.86,"uniform_accuracy_baseline":0.04987932421560751,"fixed_answer_accuracy_baseline":0.1244247787610618,"baseline_adjusted_skill":0.8401051142106327,"baseline_adjusted_score":0.8401051142106327}. Counts: {"cases": 100, "expected": 200, "succeeded": 200, "failed": 0}. Parameter-count conventions excluding lookup/embedding parameters are not model MoE active counts. No executed weight SHA is stated in this display record; exact named repository and adapter are retained.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-18 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/nlsat / test · binary_brier. Population 100 cases / 100 decisions.

Exact Cloudflare/clef; adapter clef; settings {"questions_per_call": "all-case-questions"}. Run Cloudflare__clef.

Published measured S1MB run; community execution/submission status is not assumed independent. All published diagnostics: {"binary_brier":0.34037010436766857,"accuracy":0.52,"positive_recall":0.12,"specificity":0.92,"precision":0.6,"balanced_accuracy":0.52,"f1":0.2,"true_positive":6,"false_positive":4,"false_negative":44,"true_negative":46,"positive_prevalence":0.5,"majority_accuracy_baseline":0.5,"constant_brier_baseline":0.25,"brier_skill":-0.3614804174706743,"baseline_adjusted_skill":0.040000000000000036,"baseline_adjusted_score":0.040000000000000036}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0}. Parameter-count conventions excluding lookup/embedding parameters are not model MoE active counts. No executed weight SHA is stated in this display record; exact named repository and adapter are retained.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-18 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/open_jev__amount-extraction-control-v1 / test · target_mass_at_prediction. Population 36 cases / 36 decisions.

Exact Cloudflare/clef; adapter clef; settings {"questions_per_call": "all-case-questions"}. Run Cloudflare__clef.

Published measured S1MB run; community execution/submission status is not assumed independent. Choice is target mass at the selected option; soft targets are not ordinary hard-label accuracy. All published diagnostics: {"target_mass_at_prediction":0.8611111111111112,"uniform_accuracy_baseline":0.23888888888888893,"fixed_answer_accuracy_baseline":0.2916666666666667,"baseline_adjusted_skill":0.803921568627451,"baseline_adjusted_score":0.803921568627451}. Counts: {"cases": 36, "expected": 36, "succeeded": 36, "failed": 0}. Parameter-count conventions excluding lookup/embedding parameters are not model MoE active counts. No executed weight SHA is stated in this display record; exact named repository and adapter are retained.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-18 · Samples: 36

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/open_jev__amount-extraction-control-v1 / test · binary_brier. Population 64 cases / 64 decisions.

Exact Cloudflare/clef; adapter clef; settings {"questions_per_call": "all-case-questions"}. Run Cloudflare__clef.

Published measured S1MB run; community execution/submission status is not assumed independent. All published diagnostics: {"binary_brier":0.10994812820353331,"accuracy":0.859375,"positive_recall":0.71875,"specificity":1,"precision":1,"balanced_accuracy":0.859375,"f1":0.8363636363636363,"true_positive":23,"false_positive":0,"false_negative":9,"true_negative":32,"positive_prevalence":0.5,"majority_accuracy_baseline":0.5,"constant_brier_baseline":0.25,"brier_skill":0.5602074871858668,"baseline_adjusted_skill":0.71875,"baseline_adjusted_score":0.71875}. Counts: {"cases": 64, "expected": 64, "succeeded": 64, "failed": 0}. Parameter-count conventions excluding lookup/embedding parameters are not model MoE active counts. No executed weight SHA is stated in this display record; exact named repository and adapter are retained.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-18 · Samples: 64

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/open_jev__browser-control-v1 / test · target_mass_at_prediction. Population 100 cases / 100 decisions.

Exact Cloudflare/clef; adapter clef; settings {"questions_per_call": "all-case-questions"}. Run Cloudflare__clef.

Published measured S1MB run; community execution/submission status is not assumed independent. Choice is target mass at the selected option; soft targets are not ordinary hard-label accuracy. All published diagnostics: {"target_mass_at_prediction":1,"uniform_accuracy_baseline":0.3333333333333329,"fixed_answer_accuracy_baseline":0.54,"baseline_adjusted_skill":1,"baseline_adjusted_score":1}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0}. Parameter-count conventions excluding lookup/embedding parameters are not model MoE active counts. No executed weight SHA is stated in this display record; exact named repository and adapter are retained.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-18 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/open_jev__citation-control-v1 / test · target_mass_at_prediction. Population 100 cases / 100 decisions.

Exact Cloudflare/clef; adapter clef; settings {"questions_per_call": "all-case-questions"}. Run Cloudflare__clef.

Published measured S1MB run; community execution/submission status is not assumed independent. Choice is target mass at the selected option; soft targets are not ordinary hard-label accuracy. All published diagnostics: {"target_mass_at_prediction":0.96,"uniform_accuracy_baseline":0.3333333333333329,"fixed_answer_accuracy_baseline":0.34,"baseline_adjusted_skill":0.9393939393939393,"baseline_adjusted_score":0.9393939393939393}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0}. Parameter-count conventions excluding lookup/embedding parameters are not model MoE active counts. No executed weight SHA is stated in this display record; exact named repository and adapter are retained.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-18 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/open_jev__context-retention-control-v1 / test · binary_brier. Population 100 cases / 100 decisions.

Exact Cloudflare/clef; adapter clef; settings {"questions_per_call": "all-case-questions"}. Run Cloudflare__clef.

Published measured S1MB run; community execution/submission status is not assumed independent. All published diagnostics: {"binary_brier":0.025942070533230686,"accuracy":0.96,"positive_recall":0.92,"specificity":1,"precision":1,"balanced_accuracy":0.96,"f1":0.9583333333333334,"true_positive":46,"false_positive":0,"false_negative":4,"true_negative":50,"positive_prevalence":0.5,"majority_accuracy_baseline":0.5,"constant_brier_baseline":0.25,"brier_skill":0.8962317178670772,"baseline_adjusted_skill":0.9199999999999999,"baseline_adjusted_score":0.9199999999999999}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0}. Parameter-count conventions excluding lookup/embedding parameters are not model MoE active counts. No executed weight SHA is stated in this display record; exact named repository and adapter are retained.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-18 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/open_jev__customer-control-v1 / test · target_mass_at_prediction. Population 17 cases / 17 decisions.

Exact Cloudflare/clef; adapter clef; settings {"questions_per_call": "all-case-questions"}. Run Cloudflare__clef.

Published measured S1MB run; community execution/submission status is not assumed independent. Choice is target mass at the selected option; soft targets are not ordinary hard-label accuracy. All published diagnostics: {"target_mass_at_prediction":0.8529411764705882,"uniform_accuracy_baseline":0.25,"fixed_answer_accuracy_baseline":0.3235294117647059,"baseline_adjusted_skill":0.7826086956521737,"baseline_adjusted_score":0.7826086956521737}. Counts: {"cases": 17, "expected": 17, "succeeded": 17, "failed": 0}. Parameter-count conventions excluding lookup/embedding parameters are not model MoE active counts. No executed weight SHA is stated in this display record; exact named repository and adapter are retained.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-18 · Samples: 17

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/open_jev__customer-control-v1 / test · binary_brier. Population 33 cases / 33 decisions.

Exact Cloudflare/clef; adapter clef; settings {"questions_per_call": "all-case-questions"}. Run Cloudflare__clef.

Published measured S1MB run; community execution/submission status is not assumed independent. All published diagnostics: {"binary_brier":0.0012536016290700806,"accuracy":1,"positive_recall":1,"specificity":1,"precision":1,"balanced_accuracy":1,"f1":1,"true_positive":16,"false_positive":0,"false_negative":0,"true_negative":17,"positive_prevalence":0.48484848484848486,"majority_accuracy_baseline":0.5151515151515151,"constant_brier_baseline":0.2497704315886135,"brier_skill":0.994980984654201,"baseline_adjusted_skill":1,"baseline_adjusted_score":1}. Counts: {"cases": 33, "expected": 33, "succeeded": 33, "failed": 0}. Parameter-count conventions excluding lookup/embedding parameters are not model MoE active counts. No executed weight SHA is stated in this display record; exact named repository and adapter are retained.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-18 · Samples: 33

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/open_jev__customer-control-v1 / test · normalized_expected_score_mae. Population 50 cases / 50 decisions.

Exact Cloudflare/clef; adapter clef; settings {"questions_per_call": "all-case-questions"}. Run Cloudflare__clef.

Published measured S1MB run; community execution/submission status is not assumed independent. All published diagnostics: {"normalized_expected_score_mae":0.1462834028340876,"normalized_score_rmse":0.29567424127089104,"constant_score_baseline":0.25,"constant_mae_baseline":0.33,"baseline_adjusted_skill":0.5567169611088254,"baseline_adjusted_score":0.5567169611088254}. Counts: {"cases": 50, "expected": 50, "succeeded": 50, "failed": 0}. Parameter-count conventions excluding lookup/embedding parameters are not model MoE active counts. No executed weight SHA is stated in this display record; exact named repository and adapter are retained.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-18 · Samples: 50

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/open_jev__drone-control-v1 / test · target_mass_at_prediction. Population 32 cases / 32 decisions.

Exact Cloudflare/clef; adapter clef; settings {"questions_per_call": "all-case-questions"}. Run Cloudflare__clef.

Published measured S1MB run; community execution/submission status is not assumed independent. Choice is target mass at the selected option; soft targets are not ordinary hard-label accuracy. All published diagnostics: {"target_mass_at_prediction":0.21875,"uniform_accuracy_baseline":0.17812500000000006,"fixed_answer_accuracy_baseline":0.2875,"baseline_adjusted_skill":-0.0964912280701754,"baseline_adjusted_score":0}. Counts: {"cases": 32, "expected": 32, "succeeded": 32, "failed": 0}. Parameter-count conventions excluding lookup/embedding parameters are not model MoE active counts. No executed weight SHA is stated in this display record; exact named repository and adapter are retained.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-18 · Samples: 32

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/open_jev__drone-control-v1 / test · binary_brier. Population 34 cases / 34 decisions.

Exact Cloudflare/clef; adapter clef; settings {"questions_per_call": "all-case-questions"}. Run Cloudflare__clef.

Published measured S1MB run; community execution/submission status is not assumed independent. All published diagnostics: {"binary_brier":0.051266015401809695,"accuracy":0.9117647058823529,"positive_recall":0.8235294117647058,"specificity":1,"precision":1,"balanced_accuracy":0.9117647058823529,"f1":0.9032258064516129,"true_positive":14,"false_positive":0,"false_negative":3,"true_negative":17,"positive_prevalence":0.5,"majority_accuracy_baseline":0.5,"constant_brier_baseline":0.25,"brier_skill":0.7949359383927612,"baseline_adjusted_skill":0.8235294117647058,"baseline_adjusted_score":0.8235294117647058}. Counts: {"cases": 34, "expected": 34, "succeeded": 34, "failed": 0}. Parameter-count conventions excluding lookup/embedding parameters are not model MoE active counts. No executed weight SHA is stated in this display record; exact named repository and adapter are retained.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-18 · Samples: 34

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/open_jev__drone-control-v1 / test · normalized_expected_score_mae. Population 34 cases / 34 decisions.

Exact Cloudflare/clef; adapter clef; settings {"questions_per_call": "all-case-questions"}. Run Cloudflare__clef.

Published measured S1MB run; community execution/submission status is not assumed independent. All published diagnostics: {"normalized_expected_score_mae":0.4088315679143895,"normalized_score_rmse":0.47222510646032284,"constant_score_baseline":0.5,"constant_mae_baseline":0.14705882352941177,"baseline_adjusted_skill":-1.7800546618178488,"baseline_adjusted_score":0}. Counts: {"cases": 34, "expected": 34, "succeeded": 34, "failed": 0}. Parameter-count conventions excluding lookup/embedding parameters are not model MoE active counts. No executed weight SHA is stated in this display record; exact named repository and adapter are retained.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-18 · Samples: 34

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/open_jev__email-selection-control-v1 / test · target_mass_at_prediction. Population 48 cases / 48 decisions.

Exact Cloudflare/clef; adapter clef; settings {"questions_per_call": "all-case-questions"}. Run Cloudflare__clef.

Published measured S1MB run; community execution/submission status is not assumed independent. Choice is target mass at the selected option; soft targets are not ordinary hard-label accuracy. All published diagnostics: {"target_mass_at_prediction":1,"uniform_accuracy_baseline":0.13194444444444448,"fixed_answer_accuracy_baseline":0.6006944444444444,"baseline_adjusted_skill":1,"baseline_adjusted_score":1}. Counts: {"cases": 48, "expected": 48, "succeeded": 48, "failed": 0}. Parameter-count conventions excluding lookup/embedding parameters are not model MoE active counts. No executed weight SHA is stated in this display record; exact named repository and adapter are retained.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-18 · Samples: 48

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/open_jev__email-selection-control-v1 / test · binary_brier. Population 52 cases / 52 decisions.

Exact Cloudflare/clef; adapter clef; settings {"questions_per_call": "all-case-questions"}. Run Cloudflare__clef.

Published measured S1MB run; community execution/submission status is not assumed independent. All published diagnostics: {"binary_brier":0.00043130299170984505,"accuracy":1,"positive_recall":1,"specificity":1,"precision":1,"balanced_accuracy":1,"f1":1,"true_positive":26,"false_positive":0,"false_negative":0,"true_negative":26,"positive_prevalence":0.5,"majority_accuracy_baseline":0.5,"constant_brier_baseline":0.25,"brier_skill":0.9982747880331606,"baseline_adjusted_skill":1,"baseline_adjusted_score":1}. Counts: {"cases": 52, "expected": 52, "succeeded": 52, "failed": 0}. Parameter-count conventions excluding lookup/embedding parameters are not model MoE active counts. No executed weight SHA is stated in this display record; exact named repository and adapter are retained.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-18 · Samples: 52

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/open_jev__entity-alignment-control-v1 / test · binary_brier. Population 75 cases / 75 decisions.

Exact Cloudflare/clef; adapter clef; settings {"questions_per_call": "all-case-questions"}. Run Cloudflare__clef.

Published measured S1MB run; community execution/submission status is not assumed independent. All published diagnostics: {"binary_brier":0.008947317585563145,"accuracy":1,"positive_recall":1,"specificity":1,"precision":1,"balanced_accuracy":1,"f1":1,"true_positive":39,"false_positive":0,"false_negative":0,"true_negative":36,"positive_prevalence":0.52,"majority_accuracy_baseline":0.52,"constant_brier_baseline":0.24959999999999993,"brier_skill":0.9641533750578399,"baseline_adjusted_skill":1,"baseline_adjusted_score":1}. Counts: {"cases": 75, "expected": 75, "succeeded": 75, "failed": 0}. Parameter-count conventions excluding lookup/embedding parameters are not model MoE active counts. No executed weight SHA is stated in this display record; exact named repository and adapter are retained.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-18 · Samples: 75

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/open_jev__entity-alignment-control-v1 / test · normalized_expected_score_mae. Population 25 cases / 25 decisions.

Exact Cloudflare/clef; adapter clef; settings {"questions_per_call": "all-case-questions"}. Run Cloudflare__clef.

Published measured S1MB run; community execution/submission status is not assumed independent. All published diagnostics: {"normalized_expected_score_mae":0.16874201072379946,"normalized_score_rmse":0.2325605298800104,"constant_score_baseline":0.5,"constant_mae_baseline":0.28,"baseline_adjusted_skill":0.39734996170071624,"baseline_adjusted_score":0.39734996170071624}. Counts: {"cases": 25, "expected": 25, "succeeded": 25, "failed": 0}. Parameter-count conventions excluding lookup/embedding parameters are not model MoE active counts. No executed weight SHA is stated in this display record; exact named repository and adapter are retained.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-18 · Samples: 25

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/open_jev__ir-control-v1 / test · target_mass_at_prediction. Population 17 cases / 17 decisions.

Exact Cloudflare/clef; adapter clef; settings {"questions_per_call": "all-case-questions"}. Run Cloudflare__clef.

Published measured S1MB run; community execution/submission status is not assumed independent. Choice is target mass at the selected option; soft targets are not ordinary hard-label accuracy. All published diagnostics: {"target_mass_at_prediction":0.8823529411764706,"uniform_accuracy_baseline":0.25,"fixed_answer_accuracy_baseline":0.2647058823529412,"baseline_adjusted_skill":0.84,"baseline_adjusted_score":0.84}. Counts: {"cases": 17, "expected": 17, "succeeded": 17, "failed": 0}. Parameter-count conventions excluding lookup/embedding parameters are not model MoE active counts. No executed weight SHA is stated in this display record; exact named repository and adapter are retained.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-18 · Samples: 17

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/open_jev__ir-control-v1 / test · binary_brier. Population 28 cases / 28 decisions.

Exact Cloudflare/clef; adapter clef; settings {"questions_per_call": "all-case-questions"}. Run Cloudflare__clef.

Published measured S1MB run; community execution/submission status is not assumed independent. All published diagnostics: {"binary_brier":0.06404409169823458,"accuracy":0.9285714285714286,"positive_recall":1,"specificity":0.8571428571428571,"precision":0.875,"balanced_accuracy":0.9285714285714286,"f1":0.9333333333333333,"true_positive":14,"false_positive":2,"false_negative":0,"true_negative":12,"positive_prevalence":0.5,"majority_accuracy_baseline":0.5,"constant_brier_baseline":0.25,"brier_skill":0.7438236332070617,"baseline_adjusted_skill":0.8571428571428572,"baseline_adjusted_score":0.8571428571428572}. Counts: {"cases": 28, "expected": 28, "succeeded": 28, "failed": 0}. Parameter-count conventions excluding lookup/embedding parameters are not model MoE active counts. No executed weight SHA is stated in this display record; exact named repository and adapter are retained.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-18 · Samples: 28

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/open_jev__ir-control-v1 / test · normalized_expected_score_mae. Population 55 cases / 55 decisions.

Exact Cloudflare/clef; adapter clef; settings {"questions_per_call": "all-case-questions"}. Run Cloudflare__clef.

Published measured S1MB run; community execution/submission status is not assumed independent. All published diagnostics: {"normalized_expected_score_mae":0.07737521357890782,"normalized_score_rmse":0.11908367870445373,"constant_score_baseline":0.3333333333333333,"constant_mae_baseline":0.16969696969696968,"baseline_adjusted_skill":0.5440389199814359,"baseline_adjusted_score":0.5440389199814359}. Counts: {"cases": 55, "expected": 55, "succeeded": 55, "failed": 0}. Parameter-count conventions excluding lookup/embedding parameters are not model MoE active counts. No executed weight SHA is stated in this display record; exact named repository and adapter are retained.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-18 · Samples: 55

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/open_jev__mailroom-control-v1 / test · target_mass_at_prediction. Population 15 cases / 15 decisions.

Exact Cloudflare/clef; adapter clef; settings {"questions_per_call": "all-case-questions"}. Run Cloudflare__clef.

Published measured S1MB run; community execution/submission status is not assumed independent. Choice is target mass at the selected option; soft targets are not ordinary hard-label accuracy. All published diagnostics: {"target_mass_at_prediction":1,"uniform_accuracy_baseline":0.15555555555555553,"fixed_answer_accuracy_baseline":0.2,"baseline_adjusted_skill":1,"baseline_adjusted_score":1}. Counts: {"cases": 15, "expected": 15, "succeeded": 15, "failed": 0}. Parameter-count conventions excluding lookup/embedding parameters are not model MoE active counts. No executed weight SHA is stated in this display record; exact named repository and adapter are retained.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-18 · Samples: 15

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/open_jev__mailroom-control-v1 / test · binary_brier. Population 85 cases / 85 decisions.

Exact Cloudflare/clef; adapter clef; settings {"questions_per_call": "all-case-questions"}. Run Cloudflare__clef.

Published measured S1MB run; community execution/submission status is not assumed independent. All published diagnostics: {"binary_brier":0.03738835691620036,"accuracy":0.9529411764705882,"positive_recall":0.9555555555555556,"specificity":0.95,"precision":0.9555555555555556,"balanced_accuracy":0.9527777777777777,"f1":0.9555555555555556,"true_positive":43,"false_positive":2,"false_negative":2,"true_negative":38,"positive_prevalence":0.5294117647058824,"majority_accuracy_baseline":0.5294117647058824,"constant_brier_baseline":0.24913494809688563,"brier_skill":0.8499272896002512,"baseline_adjusted_skill":0.9055555555555554,"baseline_adjusted_score":0.9055555555555554}. Counts: {"cases": 85, "expected": 85, "succeeded": 85, "failed": 0}. Parameter-count conventions excluding lookup/embedding parameters are not model MoE active counts. No executed weight SHA is stated in this display record; exact named repository and adapter are retained.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-18 · Samples: 85

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/open_jev__painting-geometry-v1 / test · target_mass_at_prediction. Population 25 cases / 25 decisions.

Exact Cloudflare/clef; adapter clef; settings {"questions_per_call": "all-case-questions"}. Run Cloudflare__clef.

Published measured S1MB run; community execution/submission status is not assumed independent. Choice is target mass at the selected option; soft targets are not ordinary hard-label accuracy. All published diagnostics: {"target_mass_at_prediction":1,"uniform_accuracy_baseline":0.07805555555555556,"fixed_answer_accuracy_baseline":0.24,"baseline_adjusted_skill":1,"baseline_adjusted_score":1}. Counts: {"cases": 25, "expected": 25, "succeeded": 25, "failed": 0}. Parameter-count conventions excluding lookup/embedding parameters are not model MoE active counts. No executed weight SHA is stated in this display record; exact named repository and adapter are retained.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-18 · Samples: 25

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/open_jev__painting-geometry-v1 / test · binary_brier. Population 50 cases / 50 decisions.

Exact Cloudflare/clef; adapter clef; settings {"questions_per_call": "all-case-questions"}. Run Cloudflare__clef.

Published measured S1MB run; community execution/submission status is not assumed independent. All published diagnostics: {"binary_brier":0.2752252984813446,"accuracy":0.7,"positive_recall":0.5714285714285714,"specificity":1,"precision":1,"balanced_accuracy":0.7857142857142857,"f1":0.7272727272727273,"true_positive":20,"false_positive":0,"false_negative":15,"true_negative":15,"positive_prevalence":0.7,"majority_accuracy_baseline":0.7,"constant_brier_baseline":0.20999999999999996,"brier_skill":-0.3105966594349745,"baseline_adjusted_skill":0.5714285714285714,"baseline_adjusted_score":0.5714285714285714}. Counts: {"cases": 50, "expected": 50, "succeeded": 50, "failed": 0}. Parameter-count conventions excluding lookup/embedding parameters are not model MoE active counts. No executed weight SHA is stated in this display record; exact named repository and adapter are retained.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-18 · Samples: 50

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/open_jev__painting-geometry-v1 / test · normalized_expected_score_mae. Population 25 cases / 25 decisions.

Exact Cloudflare/clef; adapter clef; settings {"questions_per_call": "all-case-questions"}. Run Cloudflare__clef.

Published measured S1MB run; community execution/submission status is not assumed independent. All published diagnostics: {"normalized_expected_score_mae":0.048326771324500445,"normalized_score_rmse":0.10264411673900314,"constant_score_baseline":0.5,"constant_mae_baseline":0.28,"baseline_adjusted_skill":0.8274043881267841,"baseline_adjusted_score":0.8274043881267841}. Counts: {"cases": 25, "expected": 25, "succeeded": 25, "failed": 0}. Parameter-count conventions excluding lookup/embedding parameters are not model MoE active counts. No executed weight SHA is stated in this display record; exact named repository and adapter are retained.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-18 · Samples: 25

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/open_jev__phone-extraction-control-v1 / test · target_mass_at_prediction. Population 79 cases / 79 decisions.

Exact Cloudflare/clef; adapter clef; settings {"questions_per_call": "all-case-questions"}. Run Cloudflare__clef.

Published measured S1MB run; community execution/submission status is not assumed independent. Choice is target mass at the selected option; soft targets are not ordinary hard-label accuracy. All published diagnostics: {"target_mass_at_prediction":0.7974683544303798,"uniform_accuracy_baseline":0.24240506329113914,"fixed_answer_accuracy_baseline":0.4177215189873418,"baseline_adjusted_skill":0.6521739130434783,"baseline_adjusted_score":0.6521739130434783}. Counts: {"cases": 79, "expected": 79, "succeeded": 79, "failed": 0}. Parameter-count conventions excluding lookup/embedding parameters are not model MoE active counts. No executed weight SHA is stated in this display record; exact named repository and adapter are retained.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-18 · Samples: 79

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/open_jev__phone-extraction-control-v1 / test · binary_brier. Population 21 cases / 21 decisions.

Exact Cloudflare/clef; adapter clef; settings {"questions_per_call": "all-case-questions"}. Run Cloudflare__clef.

Published measured S1MB run; community execution/submission status is not assumed independent. All published diagnostics: {"binary_brier":0.00020390039945117444,"accuracy":1,"positive_recall":1,"specificity":1,"precision":1,"balanced_accuracy":1,"f1":1,"true_positive":11,"false_positive":0,"false_negative":0,"true_negative":10,"positive_prevalence":0.5238095238095238,"majority_accuracy_baseline":0.5238095238095238,"constant_brier_baseline":0.24943310657596363,"brier_skill":0.9991825447622003,"baseline_adjusted_skill":1,"baseline_adjusted_score":1}. Counts: {"cases": 21, "expected": 21, "succeeded": 21, "failed": 0}. Parameter-count conventions excluding lookup/embedding parameters are not model MoE active counts. No executed weight SHA is stated in this display record; exact named repository and adapter are retained.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-18 · Samples: 21

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/open_jev__reasoning-control-v1 / test · target_mass_at_prediction. Population 33 cases / 33 decisions.

Exact Cloudflare/clef; adapter clef; settings {"questions_per_call": "all-case-questions"}. Run Cloudflare__clef.

Published measured S1MB run; community execution/submission status is not assumed independent. Choice is target mass at the selected option; soft targets are not ordinary hard-label accuracy. All published diagnostics: {"target_mass_at_prediction":0.8181818181818182,"uniform_accuracy_baseline":0.3237373737373738,"fixed_answer_accuracy_baseline":0.3939393939393939,"baseline_adjusted_skill":0.7000000000000001,"baseline_adjusted_score":0.7000000000000001}. Counts: {"cases": 33, "expected": 33, "succeeded": 33, "failed": 0}. Parameter-count conventions excluding lookup/embedding parameters are not model MoE active counts. No executed weight SHA is stated in this display record; exact named repository and adapter are retained.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-18 · Samples: 33

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/open_jev__reasoning-control-v1 / test · binary_brier. Population 33 cases / 33 decisions.

Exact Cloudflare/clef; adapter clef; settings {"questions_per_call": "all-case-questions"}. Run Cloudflare__clef.

Published measured S1MB run; community execution/submission status is not assumed independent. All published diagnostics: {"binary_brier":0.23815746838303076,"accuracy":0.6363636363636364,"positive_recall":0.35294117647058826,"specificity":0.9375,"precision":0.8571428571428571,"balanced_accuracy":0.6452205882352942,"f1":0.5,"true_positive":6,"false_positive":1,"false_negative":11,"true_negative":15,"positive_prevalence":0.5151515151515151,"majority_accuracy_baseline":0.5151515151515151,"constant_brier_baseline":0.2497704315886135,"brier_skill":0.0464945475399986,"baseline_adjusted_skill":0.2904411764705883,"baseline_adjusted_score":0.2904411764705883}. Counts: {"cases": 33, "expected": 33, "succeeded": 33, "failed": 0}. Parameter-count conventions excluding lookup/embedding parameters are not model MoE active counts. No executed weight SHA is stated in this display record; exact named repository and adapter are retained.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-18 · Samples: 33

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/open_jev__reasoning-control-v1 / test · normalized_expected_score_mae. Population 34 cases / 34 decisions.

Exact Cloudflare/clef; adapter clef; settings {"questions_per_call": "all-case-questions"}. Run Cloudflare__clef.

Published measured S1MB run; community execution/submission status is not assumed independent. All published diagnostics: {"normalized_expected_score_mae":0.13454057040907372,"normalized_score_rmse":0.22456241826594495,"constant_score_baseline":0.7083333333333333,"constant_mae_baseline":0.34313725490196084,"baseline_adjusted_skill":0.6079103376649853,"baseline_adjusted_score":0.6079103376649853}. Counts: {"cases": 34, "expected": 34, "succeeded": 34, "failed": 0}. Parameter-count conventions excluding lookup/embedding parameters are not model MoE active counts. No executed weight SHA is stated in this display record; exact named repository and adapter are retained.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-18 · Samples: 34

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/open_jev__silent-failure-control-v1 / test · binary_brier. Population 100 cases / 100 decisions.

Exact Cloudflare/clef; adapter clef; settings {"questions_per_call": "all-case-questions"}. Run Cloudflare__clef.

Published measured S1MB run; community execution/submission status is not assumed independent. All published diagnostics: {"binary_brier":0.04178185988815725,"accuracy":0.94,"positive_recall":0.96,"specificity":0.92,"precision":0.9230769230769231,"balanced_accuracy":0.94,"f1":0.9411764705882353,"true_positive":48,"false_positive":4,"false_negative":2,"true_negative":46,"positive_prevalence":0.5,"majority_accuracy_baseline":0.5,"constant_brier_baseline":0.25,"brier_skill":0.832872560447371,"baseline_adjusted_skill":0.8799999999999999,"baseline_adjusted_score":0.8799999999999999}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0}. Parameter-count conventions excluding lookup/embedding parameters are not model MoE active counts. No executed weight SHA is stated in this display record; exact named repository and adapter are retained.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-18 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/open_jev__snake-v1 / test · target_mass_at_prediction. Population 25 cases / 25 decisions.

Exact Cloudflare/clef; adapter clef; settings {"questions_per_call": "all-case-questions"}. Run Cloudflare__clef.

Published measured S1MB run; community execution/submission status is not assumed independent. Choice is target mass at the selected option; soft targets are not ordinary hard-label accuracy. All published diagnostics: {"target_mass_at_prediction":0.4133333333333333,"uniform_accuracy_baseline":0.3333333333333332,"fixed_answer_accuracy_baseline":0.35333333333333333,"baseline_adjusted_skill":0.09278350515463908,"baseline_adjusted_score":0.09278350515463908}. Counts: {"cases": 25, "expected": 25, "succeeded": 25, "failed": 0}. Parameter-count conventions excluding lookup/embedding parameters are not model MoE active counts. No executed weight SHA is stated in this display record; exact named repository and adapter are retained.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-18 · Samples: 25

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/open_jev__snake-v1 / test · binary_brier. Population 75 cases / 75 decisions.

Exact Cloudflare/clef; adapter clef; settings {"questions_per_call": "all-case-questions"}. Run Cloudflare__clef.

Published measured S1MB run; community execution/submission status is not assumed independent. All published diagnostics: {"binary_brier":0.2087606330142149,"accuracy":0.6533333333333333,"positive_recall":0.5405405405405406,"specificity":0.7631578947368421,"precision":0.6896551724137931,"balanced_accuracy":0.6518492176386914,"f1":0.6060606060606061,"true_positive":20,"false_positive":9,"false_negative":17,"true_negative":29,"positive_prevalence":0.49333333333333335,"majority_accuracy_baseline":0.5066666666666666,"constant_brier_baseline":0.24995555555555535,"brier_skill":0.16480898954128032,"baseline_adjusted_skill":0.3036984352773828,"baseline_adjusted_score":0.3036984352773828}. Counts: {"cases": 75, "expected": 75, "succeeded": 75, "failed": 0}. Parameter-count conventions excluding lookup/embedding parameters are not model MoE active counts. No executed weight SHA is stated in this display record; exact named repository and adapter are retained.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-18 · Samples: 75

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/open_jev__sponsor-segment-control-v1 / test · target_mass_at_prediction. Population 100 cases / 100 decisions.

Exact Cloudflare/clef; adapter clef; settings {"questions_per_call": "all-case-questions"}. Run Cloudflare__clef.

Published measured S1MB run; community execution/submission status is not assumed independent. Choice is target mass at the selected option; soft targets are not ordinary hard-label accuracy. All published diagnostics: {"target_mass_at_prediction":0.98,"uniform_accuracy_baseline":0.1428571428571427,"fixed_answer_accuracy_baseline":0.22,"baseline_adjusted_skill":0.9743589743589743,"baseline_adjusted_score":0.9743589743589743}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0}. Parameter-count conventions excluding lookup/embedding parameters are not model MoE active counts. No executed weight SHA is stated in this display record; exact named repository and adapter are retained.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-18 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/open_jev__tic_tac_toe-v1 / test · target_mass_at_prediction. Population 100 cases / 100 decisions.

Exact Cloudflare/clef; adapter clef; settings {"questions_per_call": "all-case-questions"}. Run Cloudflare__clef.

Published measured S1MB run; community execution/submission status is not assumed independent. Choice is target mass at the selected option; soft targets are not ordinary hard-label accuracy. All published diagnostics: {"target_mass_at_prediction":0.375,"uniform_accuracy_baseline":0.32416666666666627,"fixed_answer_accuracy_baseline":0.34,"baseline_adjusted_skill":0.053030303030303,"baseline_adjusted_score":0.053030303030303}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0}. Parameter-count conventions excluding lookup/embedding parameters are not model MoE active counts. No executed weight SHA is stated in this display record; exact named repository and adapter are retained.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-18 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/open_jev__vizdoom-basic-v1 / test · binary_brier. Population 33 cases / 33 decisions.

Exact Cloudflare/clef; adapter clef; settings {"questions_per_call": "all-case-questions"}. Run Cloudflare__clef.

Published measured S1MB run; community execution/submission status is not assumed independent. All published diagnostics: {"binary_brier":0.01971391215988438,"accuracy":0.9696969696969697,"positive_recall":1,"specificity":0.9411764705882353,"precision":0.9411764705882353,"balanced_accuracy":0.9705882352941176,"f1":0.9696969696969697,"true_positive":16,"false_positive":1,"false_negative":0,"true_negative":16,"positive_prevalence":0.48484848484848486,"majority_accuracy_baseline":0.5151515151515151,"constant_brier_baseline":0.2497704315886135,"brier_skill":0.9210718737422277,"baseline_adjusted_skill":0.9411764705882353,"baseline_adjusted_score":0.9411764705882353}. Counts: {"cases": 33, "expected": 33, "succeeded": 33, "failed": 0}. Parameter-count conventions excluding lookup/embedding parameters are not model MoE active counts. No executed weight SHA is stated in this display record; exact named repository and adapter are retained.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-18 · Samples: 33

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/open_jev__vizdoom-basic-v1 / test · normalized_expected_score_mae. Population 33 cases / 33 decisions.

Exact Cloudflare/clef; adapter clef; settings {"questions_per_call": "all-case-questions"}. Run Cloudflare__clef.

Published measured S1MB run; community execution/submission status is not assumed independent. All published diagnostics: {"normalized_expected_score_mae":0.061528103200323654,"normalized_score_rmse":0.07563651595767494,"constant_score_baseline":0.5,"constant_mae_baseline":0.2196969696969697,"baseline_adjusted_skill":0.7199410475019751,"baseline_adjusted_score":0.7199410475019751}. Counts: {"cases": 33, "expected": 33, "succeeded": 33, "failed": 0}. Parameter-count conventions excluding lookup/embedding parameters are not model MoE active counts. No executed weight SHA is stated in this display record; exact named repository and adapter are retained.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-18 · Samples: 33

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/open_jev__workflow-controls-v1__agent_trace_observability / test · binary_brier. Population 100 cases / 100 decisions.

Exact Cloudflare/clef; adapter clef; settings {"questions_per_call": "all-case-questions"}. Run Cloudflare__clef.

Published measured S1MB run; community execution/submission status is not assumed independent. All published diagnostics: {"binary_brier":0.12441047728393965,"accuracy":0.84,"positive_recall":0.8181818181818182,"specificity":0.8461538461538461,"precision":0.6,"balanced_accuracy":0.8321678321678322,"f1":0.6923076923076923,"true_positive":18,"false_positive":12,"false_negative":4,"true_negative":66,"positive_prevalence":0.22,"majority_accuracy_baseline":0.78,"constant_brier_baseline":0.17160000000000028,"brier_skill":0.27499721862506143,"baseline_adjusted_skill":0.6643356643356644,"baseline_adjusted_score":0.6643356643356644}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0}. Parameter-count conventions excluding lookup/embedding parameters are not model MoE active counts. No executed weight SHA is stated in this display record; exact named repository and adapter are retained.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-18 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/open_jev__workflow-controls-v1__customer_service / test · binary_brier. Population 100 cases / 100 decisions.

Exact Cloudflare/clef; adapter clef; settings {"questions_per_call": "all-case-questions"}. Run Cloudflare__clef.

Published measured S1MB run; community execution/submission status is not assumed independent. All published diagnostics: {"binary_brier":0.12815454914443714,"accuracy":0.85,"positive_recall":0.6216216216216216,"specificity":0.9841269841269841,"precision":0.9583333333333334,"balanced_accuracy":0.8028743028743028,"f1":0.7540983606557377,"true_positive":23,"false_positive":1,"false_negative":14,"true_negative":62,"positive_prevalence":0.37,"majority_accuracy_baseline":0.63,"constant_brier_baseline":0.23310000000000008,"brier_skill":0.45021643438679926,"baseline_adjusted_skill":0.6057486057486057,"baseline_adjusted_score":0.6057486057486057}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0}. Parameter-count conventions excluding lookup/embedding parameters are not model MoE active counts. No executed weight SHA is stated in this display record; exact named repository and adapter are retained.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-18 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/open_jev__workflow-controls-v1__invoice_processing / test · binary_brier. Population 100 cases / 100 decisions.

Exact Cloudflare/clef; adapter clef; settings {"questions_per_call": "all-case-questions"}. Run Cloudflare__clef.

Published measured S1MB run; community execution/submission status is not assumed independent. All published diagnostics: {"binary_brier":0.057216465390855946,"accuracy":0.93,"positive_recall":0.8205128205128205,"specificity":1,"precision":1,"balanced_accuracy":0.9102564102564102,"f1":0.9014084507042254,"true_positive":32,"false_positive":0,"false_negative":7,"true_negative":61,"positive_prevalence":0.39,"majority_accuracy_baseline":0.61,"constant_brier_baseline":0.2379000000000002,"brier_skill":0.7594936301351161,"baseline_adjusted_skill":0.8205128205128205,"baseline_adjusted_score":0.8205128205128205}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0}. Parameter-count conventions excluding lookup/embedding parameters are not model MoE active counts. No executed weight SHA is stated in this display record; exact named repository and adapter are retained.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-18 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/open_jev__workflow-controls-v1__security_incidents / test · binary_brier. Population 100 cases / 100 decisions.

Exact Cloudflare/clef; adapter clef; settings {"questions_per_call": "all-case-questions"}. Run Cloudflare__clef.

Published measured S1MB run; community execution/submission status is not assumed independent. All published diagnostics: {"binary_brier":0.05260487279120984,"accuracy":0.93,"positive_recall":0.8387096774193549,"specificity":0.9710144927536232,"precision":0.9285714285714286,"balanced_accuracy":0.904862085086489,"f1":0.8813559322033898,"true_positive":26,"false_positive":2,"false_negative":5,"true_negative":67,"positive_prevalence":0.31,"majority_accuracy_baseline":0.69,"constant_brier_baseline":0.21389999999999987,"brier_skill":0.754067915889622,"baseline_adjusted_skill":0.809724170172978,"baseline_adjusted_score":0.809724170172978}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0}. Parameter-count conventions excluding lookup/embedding parameters are not model MoE active counts. No executed weight SHA is stated in this display record; exact named repository and adapter are retained.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-18 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/openbookqa / test · target_mass_at_prediction. Population 100 cases / 100 decisions.

Exact Cloudflare/clef; adapter clef; settings {"questions_per_call": "all-case-questions"}. Run Cloudflare__clef.

Published measured S1MB run; community execution/submission status is not assumed independent. Choice is target mass at the selected option; soft targets are not ordinary hard-label accuracy. All published diagnostics: {"target_mass_at_prediction":0.95,"uniform_accuracy_baseline":0.25,"fixed_answer_accuracy_baseline":0.28,"baseline_adjusted_skill":0.9305555555555555,"baseline_adjusted_score":0.9305555555555555}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0}. Parameter-count conventions excluding lookup/embedding parameters are not model MoE active counts. No executed weight SHA is stated in this display record; exact named repository and adapter are retained.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-18 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/patent_similarity / test · normalized_expected_score_mae. Population 100 cases / 100 decisions.

Exact Cloudflare/clef; adapter clef; settings {"questions_per_call": "all-case-questions"}. Run Cloudflare__clef.

Published measured S1MB run; community execution/submission status is not assumed independent. All published diagnostics: {"normalized_expected_score_mae":0.1423316943878308,"normalized_score_rmse":0.1744531871583556,"constant_score_baseline":0.25,"constant_mae_baseline":0.2325,"baseline_adjusted_skill":0.38782066929965253,"baseline_adjusted_score":0.38782066929965253}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0}. Parameter-count conventions excluding lookup/embedding parameters are not model MoE active counts. No executed weight SHA is stated in this display record; exact named repository and adapter are retained.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-18 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/paws / test · binary_brier. Population 100 cases / 100 decisions.

Exact Cloudflare/clef; adapter clef; settings {"questions_per_call": "all-case-questions"}. Run Cloudflare__clef.

Published measured S1MB run; community execution/submission status is not assumed independent. All published diagnostics: {"binary_brier":0.11655353209787891,"accuracy":0.83,"positive_recall":0.92,"specificity":0.74,"precision":0.7796610169491526,"balanced_accuracy":0.8300000000000001,"f1":0.8440366972477065,"true_positive":46,"false_positive":13,"false_negative":4,"true_negative":37,"positive_prevalence":0.5,"majority_accuracy_baseline":0.5,"constant_brier_baseline":0.25,"brier_skill":0.5337858716084843,"baseline_adjusted_skill":0.6600000000000001,"baseline_adjusted_score":0.6600000000000001}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0}. Parameter-count conventions excluding lookup/embedding parameters are not model MoE active counts. No executed weight SHA is stated in this display record; exact named repository and adapter are retained.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-18 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/plane / test · binary_brier. Population 100 cases / 100 decisions.

Exact Cloudflare/clef; adapter clef; settings {"questions_per_call": "all-case-questions"}. Run Cloudflare__clef.

Published measured S1MB run; community execution/submission status is not assumed independent. All published diagnostics: {"binary_brier":0.23926534628998666,"accuracy":0.57,"positive_recall":0.54,"specificity":0.6,"precision":0.574468085106383,"balanced_accuracy":0.5700000000000001,"f1":0.5567010309278351,"true_positive":27,"false_positive":20,"false_negative":23,"true_negative":30,"positive_prevalence":0.5,"majority_accuracy_baseline":0.5,"constant_brier_baseline":0.25,"brier_skill":0.042938614840053346,"baseline_adjusted_skill":0.14000000000000012,"baseline_adjusted_score":0.14000000000000012}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0}. Parameter-count conventions excluding lookup/embedding parameters are not model MoE active counts. No executed weight SHA is stated in this display record; exact named repository and adapter are retained.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-18 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/poem_sentiment / test · target_mass_at_prediction. Population 100 cases / 100 decisions.

Exact Cloudflare/clef; adapter clef; settings {"questions_per_call": "all-case-questions"}. Run Cloudflare__clef.

Published measured S1MB run; community execution/submission status is not assumed independent. Choice is target mass at the selected option; soft targets are not ordinary hard-label accuracy. All published diagnostics: {"target_mass_at_prediction":0.53,"uniform_accuracy_baseline":0.25,"fixed_answer_accuracy_baseline":0.65,"baseline_adjusted_skill":-0.34285714285714286,"baseline_adjusted_score":0}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0}. Parameter-count conventions excluding lookup/embedding parameters are not model MoE active counts. No executed weight SHA is stated in this display record; exact named repository and adapter are retained.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-18 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/qasper / test · binary_brier. Population 100 cases / 100 decisions.

Exact Cloudflare/clef; adapter clef; settings {"questions_per_call": "all-case-questions"}. Run Cloudflare__clef.

Published measured S1MB run; community execution/submission status is not assumed independent. All published diagnostics: {"binary_brier":0.08098570228981238,"accuracy":0.88,"positive_recall":0.88,"specificity":0.88,"precision":0.88,"balanced_accuracy":0.88,"f1":0.88,"true_positive":44,"false_positive":6,"false_negative":6,"true_negative":44,"positive_prevalence":0.5,"majority_accuracy_baseline":0.5,"constant_brier_baseline":0.25,"brier_skill":0.6760571908407504,"baseline_adjusted_skill":0.76,"baseline_adjusted_score":0.76}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0}. Parameter-count conventions excluding lookup/embedding parameters are not model MoE active counts. No executed weight SHA is stated in this display record; exact named repository and adapter are retained.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-18 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/quartz / test · target_mass_at_prediction. Population 100 cases / 100 decisions.

Exact Cloudflare/clef; adapter clef; settings {"questions_per_call": "all-case-questions"}. Run Cloudflare__clef.

Published measured S1MB run; community execution/submission status is not assumed independent. Choice is target mass at the selected option; soft targets are not ordinary hard-label accuracy. All published diagnostics: {"target_mass_at_prediction":0.9,"uniform_accuracy_baseline":0.5,"fixed_answer_accuracy_baseline":0.58,"baseline_adjusted_skill":0.761904761904762,"baseline_adjusted_score":0.761904761904762}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0}. Parameter-count conventions excluding lookup/embedding parameters are not model MoE active counts. No executed weight SHA is stated in this display record; exact named repository and adapter are retained.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-18 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/robust_lr / test · target_mass_at_prediction. Population 100 cases / 100 decisions.

Exact Cloudflare/clef; adapter clef; settings {"questions_per_call": "all-case-questions"}. Run Cloudflare__clef.

Published measured S1MB run; community execution/submission status is not assumed independent. Choice is target mass at the selected option; soft targets are not ordinary hard-label accuracy. All published diagnostics: {"target_mass_at_prediction":0.75,"uniform_accuracy_baseline":0.3333333333333329,"fixed_answer_accuracy_baseline":0.4,"baseline_adjusted_skill":0.5833333333333334,"baseline_adjusted_score":0.5833333333333334}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0}. Parameter-count conventions excluding lookup/embedding parameters are not model MoE active counts. No executed weight SHA is stated in this display record; exact named repository and adapter are retained.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-18 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/ruletaker / test · binary_brier. Population 100 cases / 100 decisions.

Exact Cloudflare/clef; adapter clef; settings {"questions_per_call": "all-case-questions"}. Run Cloudflare__clef.

Published measured S1MB run; community execution/submission status is not assumed independent. All published diagnostics: {"binary_brier":0.13301882951626473,"accuracy":0.8,"positive_recall":0.86,"specificity":0.74,"precision":0.7678571428571429,"balanced_accuracy":0.8,"f1":0.8113207547169812,"true_positive":43,"false_positive":13,"false_negative":7,"true_negative":37,"positive_prevalence":0.5,"majority_accuracy_baseline":0.5,"constant_brier_baseline":0.25,"brier_skill":0.4679246819349411,"baseline_adjusted_skill":0.6000000000000001,"baseline_adjusted_score":0.6000000000000001}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0}. Parameter-count conventions excluding lookup/embedding parameters are not model MoE active counts. No executed weight SHA is stated in this display record; exact named repository and adapter are retained.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-18 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/s1mb-generalization-contextual-choice / test · target_mass_at_prediction. Population 100 cases / 100 decisions.

Exact Cloudflare/clef; adapter clef; settings {"questions_per_call": "all-case-questions"}. Run Cloudflare__clef.

Published measured S1MB run; community execution/submission status is not assumed independent. Choice is target mass at the selected option; soft targets are not ordinary hard-label accuracy. All published diagnostics: {"target_mass_at_prediction":0.98,"uniform_accuracy_baseline":0.20851190476190457,"fixed_answer_accuracy_baseline":0.2233333333333332,"baseline_adjusted_skill":0.9742489270386266,"baseline_adjusted_score":0.9742489270386266}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0}. Parameter-count conventions excluding lookup/embedding parameters are not model MoE active counts. No executed weight SHA is stated in this display record; exact named repository and adapter are retained.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-18 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/s1mb-generalization-contextual-noul / test · binary_brier. Population 100 cases / 100 decisions.

Exact Cloudflare/clef; adapter clef; settings {"questions_per_call": "all-case-questions"}. Run Cloudflare__clef.

Published measured S1MB run; community execution/submission status is not assumed independent. All published diagnostics: {"binary_brier":0.04353440691264484,"accuracy":0.95,"positive_recall":0.92,"specificity":0.98,"precision":0.9787234042553191,"balanced_accuracy":0.95,"f1":0.9484536082474226,"true_positive":46,"false_positive":1,"false_negative":4,"true_negative":49,"positive_prevalence":0.5,"majority_accuracy_baseline":0.5,"constant_brier_baseline":0.25,"brier_skill":0.8258623723494206,"baseline_adjusted_skill":0.8999999999999999,"baseline_adjusted_score":0.8999999999999999}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0}. Parameter-count conventions excluding lookup/embedding parameters are not model MoE active counts. No executed weight SHA is stated in this display record; exact named repository and adapter are retained.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-18 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/s1mb-generalization-contextual-score / test · normalized_expected_score_mae. Population 100 cases / 100 decisions.

Exact Cloudflare/clef; adapter clef; settings {"questions_per_call": "all-case-questions"}. Run Cloudflare__clef.

Published measured S1MB run; community execution/submission status is not assumed independent. All published diagnostics: {"normalized_expected_score_mae":0.07663949115609285,"normalized_score_rmse":0.1278174299683955,"constant_score_baseline":0.5,"constant_mae_baseline":0.3,"baseline_adjusted_skill":0.7445350294796904,"baseline_adjusted_score":0.7445350294796904}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0}. Parameter-count conventions excluding lookup/embedding parameters are not model MoE active counts. No executed weight SHA is stated in this display record; exact named repository and adapter are retained.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-18 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/s1mb-generalization-diverse-choice / test · target_mass_at_prediction. Population 100 cases / 100 decisions.

Exact Cloudflare/clef; adapter clef; settings {"questions_per_call": "all-case-questions"}. Run Cloudflare__clef.

Published measured S1MB run; community execution/submission status is not assumed independent. Choice is target mass at the selected option; soft targets are not ordinary hard-label accuracy. All published diagnostics: {"target_mass_at_prediction":0.99,"uniform_accuracy_baseline":0.22851190476190467,"fixed_answer_accuracy_baseline":0.2433333333333333,"baseline_adjusted_skill":0.986784140969163,"baseline_adjusted_score":0.986784140969163}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0}. Parameter-count conventions excluding lookup/embedding parameters are not model MoE active counts. No executed weight SHA is stated in this display record; exact named repository and adapter are retained.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-18 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/s1mb-generalization-diverse-noul / test · binary_brier. Population 100 cases / 100 decisions.

Exact Cloudflare/clef; adapter clef; settings {"questions_per_call": "all-case-questions"}. Run Cloudflare__clef.

Published measured S1MB run; community execution/submission status is not assumed independent. All published diagnostics: {"binary_brier":0.023488303872686522,"accuracy":0.98,"positive_recall":0.96,"specificity":1,"precision":1,"balanced_accuracy":0.98,"f1":0.9795918367346939,"true_positive":48,"false_positive":0,"false_negative":2,"true_negative":50,"positive_prevalence":0.5,"majority_accuracy_baseline":0.5,"constant_brier_baseline":0.25,"brier_skill":0.9060467845092539,"baseline_adjusted_skill":0.96,"baseline_adjusted_score":0.96}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0}. Parameter-count conventions excluding lookup/embedding parameters are not model MoE active counts. No executed weight SHA is stated in this display record; exact named repository and adapter are retained.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-18 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/s1mb-generalization-diverse-score / test · normalized_expected_score_mae. Population 100 cases / 100 decisions.

Exact Cloudflare/clef; adapter clef; settings {"questions_per_call": "all-case-questions"}. Run Cloudflare__clef.

Published measured S1MB run; community execution/submission status is not assumed independent. All published diagnostics: {"normalized_expected_score_mae":0.05254361404501834,"normalized_score_rmse":0.08930271335214417,"constant_score_baseline":0.4222222222222222,"constant_mae_baseline":0.304222222222222,"baseline_adjusted_skill":0.82728541767525,"baseline_adjusted_score":0.82728541767525}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0}. Parameter-count conventions excluding lookup/embedding parameters are not model MoE active counts. No executed weight SHA is stated in this display record; exact named repository and adapter are retained.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-18 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/scicite / test · target_mass_at_prediction. Population 100 cases / 100 decisions.

Exact Cloudflare/clef; adapter clef; settings {"questions_per_call": "all-case-questions"}. Run Cloudflare__clef.

Published measured S1MB run; community execution/submission status is not assumed independent. Choice is target mass at the selected option; soft targets are not ordinary hard-label accuracy. All published diagnostics: {"target_mass_at_prediction":0.8,"uniform_accuracy_baseline":0.3333333333333329,"fixed_answer_accuracy_baseline":0.38,"baseline_adjusted_skill":0.6774193548387097,"baseline_adjusted_score":0.6774193548387097}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0}. Parameter-count conventions excluding lookup/embedding parameters are not model MoE active counts. No executed weight SHA is stated in this display record; exact named repository and adapter are retained.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-18 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/scitail / test · binary_brier. Population 100 cases / 100 decisions.

Exact Cloudflare/clef; adapter clef; settings {"questions_per_call": "all-case-questions"}. Run Cloudflare__clef.

Published measured S1MB run; community execution/submission status is not assumed independent. All published diagnostics: {"binary_brier":0.051339593824844786,"accuracy":0.92,"positive_recall":0.96,"specificity":0.88,"precision":0.8888888888888888,"balanced_accuracy":0.9199999999999999,"f1":0.9230769230769231,"true_positive":48,"false_positive":6,"false_negative":2,"true_negative":44,"positive_prevalence":0.5,"majority_accuracy_baseline":0.5,"constant_brier_baseline":0.25,"brier_skill":0.7946416247006208,"baseline_adjusted_skill":0.8399999999999999,"baseline_adjusted_score":0.8399999999999999}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0}. Parameter-count conventions excluding lookup/embedding parameters are not model MoE active counts. No executed weight SHA is stated in this display record; exact named repository and adapter are retained.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-18 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/scone / test · target_mass_at_prediction. Population 100 cases / 100 decisions.

Exact Cloudflare/clef; adapter clef; settings {"questions_per_call": "all-case-questions"}. Run Cloudflare__clef.

Published measured S1MB run; community execution/submission status is not assumed independent. Choice is target mass at the selected option; soft targets are not ordinary hard-label accuracy. All published diagnostics: {"target_mass_at_prediction":0.88,"uniform_accuracy_baseline":0.3333333333333329,"fixed_answer_accuracy_baseline":0.5,"baseline_adjusted_skill":0.76,"baseline_adjusted_score":0.76}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0}. Parameter-count conventions excluding lookup/embedding parameters are not model MoE active counts. No executed weight SHA is stated in this display record; exact named repository and adapter are retained.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-18 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/sdoh_nli / test · binary_brier. Population 100 cases / 100 decisions.

Exact Cloudflare/clef; adapter clef; settings {"questions_per_call": "all-case-questions"}. Run Cloudflare__clef.

Published measured S1MB run; community execution/submission status is not assumed independent. All published diagnostics: {"binary_brier":0.05332428226302327,"accuracy":0.94,"positive_recall":0.96,"specificity":0.92,"precision":0.9230769230769231,"balanced_accuracy":0.94,"f1":0.9411764705882353,"true_positive":48,"false_positive":4,"false_negative":2,"true_negative":46,"positive_prevalence":0.5,"majority_accuracy_baseline":0.5,"constant_brier_baseline":0.25,"brier_skill":0.7867028709479069,"baseline_adjusted_skill":0.8799999999999999,"baseline_adjusted_score":0.8799999999999999}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0}. Parameter-count conventions excluding lookup/embedding parameters are not model MoE active counts. No executed weight SHA is stated in this display record; exact named repository and adapter are retained.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-18 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/sgd / test · target_mass_at_prediction. Population 100 cases / 100 decisions.

Exact Cloudflare/clef; adapter clef; settings {"questions_per_call": "all-case-questions"}. Run Cloudflare__clef.

Published measured S1MB run; community execution/submission status is not assumed independent. Choice is target mass at the selected option; soft targets are not ordinary hard-label accuracy. All published diagnostics: {"target_mass_at_prediction":0.92,"uniform_accuracy_baseline":0.35333333333333317,"fixed_answer_accuracy_baseline":0.41,"baseline_adjusted_skill":0.8644067796610169,"baseline_adjusted_score":0.8644067796610169}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0}. Parameter-count conventions excluding lookup/embedding parameters are not model MoE active counts. No executed weight SHA is stated in this display record; exact named repository and adapter are retained.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-18 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/snli / test · target_mass_at_prediction. Population 100 cases / 100 decisions.

Exact Cloudflare/clef; adapter clef; settings {"questions_per_call": "all-case-questions"}. Run Cloudflare__clef.

Published measured S1MB run; community execution/submission status is not assumed independent. Choice is target mass at the selected option; soft targets are not ordinary hard-label accuracy. All published diagnostics: {"target_mass_at_prediction":0.85,"uniform_accuracy_baseline":0.3333333333333329,"fixed_answer_accuracy_baseline":0.38,"baseline_adjusted_skill":0.7580645161290323,"baseline_adjusted_score":0.7580645161290323}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0}. Parameter-count conventions excluding lookup/embedding parameters are not model MoE active counts. No executed weight SHA is stated in this display record; exact named repository and adapter are retained.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-18 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/sobaco-culture-choice / test · target_mass_at_prediction. Population 300 cases / 300 decisions.

Exact Cloudflare/clef; adapter clef; settings {"questions_per_call": "all-case-questions"}. Run Cloudflare__clef.

Published measured S1MB run; community execution/submission status is not assumed independent. Choice is target mass at the selected option; soft targets are not ordinary hard-label accuracy. All published diagnostics: {"target_mass_at_prediction":0.6433333333333333,"uniform_accuracy_baseline":0.3333333333333322,"fixed_answer_accuracy_baseline":0.3333333333333333,"baseline_adjusted_skill":0.46499999999999997,"baseline_adjusted_score":0.46499999999999997}. Counts: {"cases": 300, "expected": 300, "succeeded": 300, "failed": 0}. Parameter-count conventions excluding lookup/embedding parameters are not model MoE active counts. No executed weight SHA is stated in this display record; exact named repository and adapter are retained.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-18 · Samples: 300

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/spartqa / test · target_mass_at_prediction. Population 100 cases / 100 decisions.

Exact Cloudflare/clef; adapter clef; settings {"questions_per_call": "all-case-questions"}. Run Cloudflare__clef.

Published measured S1MB run; community execution/submission status is not assumed independent. Choice is target mass at the selected option; soft targets are not ordinary hard-label accuracy. All published diagnostics: {"target_mass_at_prediction":0.48,"uniform_accuracy_baseline":0.25,"fixed_answer_accuracy_baseline":0.38,"baseline_adjusted_skill":0.16129032258064513,"baseline_adjusted_score":0.16129032258064513}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0}. Parameter-count conventions excluding lookup/embedding parameters are not model MoE active counts. No executed weight SHA is stated in this display record; exact named repository and adapter are retained.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-18 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/stepgame / test · target_mass_at_prediction. Population 100 cases / 100 decisions.

Exact Cloudflare/clef; adapter clef; settings {"questions_per_call": "all-case-questions"}. Run Cloudflare__clef.

Published measured S1MB run; community execution/submission status is not assumed independent. Choice is target mass at the selected option; soft targets are not ordinary hard-label accuracy. All published diagnostics: {"target_mass_at_prediction":0.39,"uniform_accuracy_baseline":0.11111111111111088,"fixed_answer_accuracy_baseline":0.16,"baseline_adjusted_skill":0.27380952380952384,"baseline_adjusted_score":0.27380952380952384}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0}. Parameter-count conventions excluding lookup/embedding parameters are not model MoE active counts. No executed weight SHA is stated in this display record; exact named repository and adapter are retained.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-18 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/synthetic_relevance__nanobeir__NanoArguAna / test · binary_brier. Population 400 cases / 400 decisions.

Exact Cloudflare/clef; adapter clef; settings {"questions_per_call": "all-case-questions"}. Run Cloudflare__clef.

Published measured S1MB run; community execution/submission status is not assumed independent. All published diagnostics: {"binary_brier":0.044064133588074206,"accuracy":0.8378749999999999,"positive_recall":0.42847725774555045,"specificity":0.9336726824001234,"precision":0.6018518518518519,"balanced_accuracy":0.6810749700728369,"f1":0.5005775895263767,"true_positive":32.5,"false_positive":21.499999999999996,"false_negative":43.349999999999994,"true_negative":302.65,"positive_prevalence":0.1896249999999999,"majority_accuracy_baseline":0.8103750000000001,"constant_brier_baseline":0.07612360937500007,"brier_skill":0.4211502324987574,"baseline_adjusted_skill":0.3621499401456738,"baseline_adjusted_score":0.3621499401456738}. Counts: {"cases": 400, "expected": 400, "succeeded": 400, "failed": 0}. Parameter-count conventions excluding lookup/embedding parameters are not model MoE active counts. No executed weight SHA is stated in this display record; exact named repository and adapter are retained.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-18 · Samples: 400

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/synthetic_relevance__nanobeir__NanoArguAna / test · normalized_expected_score_mae. Population 400 cases / 400 decisions.

Exact Cloudflare/clef; adapter clef; settings {"questions_per_call": "all-case-questions"}. Run Cloudflare__clef.

Published measured S1MB run; community execution/submission status is not assumed independent. All published diagnostics: {"normalized_expected_score_mae":0.18919883417722302,"normalized_score_rmse":0.222955473987627,"constant_score_baseline":0,"constant_mae_baseline":0.21375,"baseline_adjusted_skill":0.11485925531123731,"baseline_adjusted_score":0.11485925531123731}. Counts: {"cases": 400, "expected": 400, "succeeded": 400, "failed": 0}. Parameter-count conventions excluding lookup/embedding parameters are not model MoE active counts. No executed weight SHA is stated in this display record; exact named repository and adapter are retained.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-18 · Samples: 400

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/synthetic_relevance__nanobeir__NanoDBPedia / test · binary_brier. Population 400 cases / 400 decisions.

Exact Cloudflare/clef; adapter clef; settings {"questions_per_call": "all-case-questions"}. Run Cloudflare__clef.

Published measured S1MB run; community execution/submission status is not assumed independent. All published diagnostics: {"binary_brier":0.06462827271892033,"accuracy":0.791125,"positive_recall":0.49273323956868265,"specificity":0.8996079768194989,"precision":0.6408536585365856,"balanced_accuracy":0.6961706081940908,"f1":0.5571163530347205,"true_positive":52.55000000000002,"false_positive":29.449999999999978,"false_negative":54.10000000000001,"true_negative":263.9,"positive_prevalence":0.2666249999999998,"majority_accuracy_baseline":0.7333750000000002,"constant_brier_baseline":0.11565485937499977,"brier_skill":0.4411970835624869,"baseline_adjusted_skill":0.39234121638818165,"baseline_adjusted_score":0.39234121638818165}. Counts: {"cases": 400, "expected": 400, "succeeded": 400, "failed": 0}. Parameter-count conventions excluding lookup/embedding parameters are not model MoE active counts. No executed weight SHA is stated in this display record; exact named repository and adapter are retained.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-18 · Samples: 400

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/synthetic_relevance__nanobeir__NanoDBPedia / test · normalized_expected_score_mae. Population 400 cases / 400 decisions.

Exact Cloudflare/clef; adapter clef; settings {"questions_per_call": "all-case-questions"}. Run Cloudflare__clef.

Published measured S1MB run; community execution/submission status is not assumed independent. All published diagnostics: {"normalized_expected_score_mae":0.18556035031098872,"normalized_score_rmse":0.2261595936311412,"constant_score_baseline":0.25,"constant_mae_baseline":0.26875,"baseline_adjusted_skill":0.30954288256376283,"baseline_adjusted_score":0.30954288256376283}. Counts: {"cases": 400, "expected": 400, "succeeded": 400, "failed": 0}. Parameter-count conventions excluding lookup/embedding parameters are not model MoE active counts. No executed weight SHA is stated in this display record; exact named repository and adapter are retained.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-18 · Samples: 400

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/synthetic_relevance__nanobeir__NanoFiQA2018 / test · binary_brier. Population 400 cases / 400 decisions.

Exact Cloudflare/clef; adapter clef; settings {"questions_per_call": "all-case-questions"}. Run Cloudflare__clef.

Published measured S1MB run; community execution/submission status is not assumed independent. All published diagnostics: {"binary_brier":0.04822666627733348,"accuracy":0.8166249999999998,"positive_recall":0.6639821029082772,"specificity":0.8758022549869905,"precision":0.6745454545454544,"balanced_accuracy":0.7698921789476338,"f1":0.6692220969560315,"true_positive":74.19999999999997,"false_positive":35.8,"false_negative":37.550000000000026,"true_negative":252.44999999999996,"positive_prevalence":0.2793749999999999,"majority_accuracy_baseline":0.7206250000000001,"constant_brier_baseline":0.11315585937499995,"brier_skill":0.5738031901864693,"baseline_adjusted_skill":0.5397843578952677,"baseline_adjusted_score":0.5397843578952677}. Counts: {"cases": 400, "expected": 400, "succeeded": 400, "failed": 0}. Parameter-count conventions excluding lookup/embedding parameters are not model MoE active counts. No executed weight SHA is stated in this display record; exact named repository and adapter are retained.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-18 · Samples: 400

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/synthetic_relevance__nanobeir__NanoFiQA2018 / test · normalized_expected_score_mae. Population 400 cases / 400 decisions.

Exact Cloudflare/clef; adapter clef; settings {"questions_per_call": "all-case-questions"}. Run Cloudflare__clef.

Published measured S1MB run; community execution/submission status is not assumed independent. All published diagnostics: {"normalized_expected_score_mae":0.1857174855004996,"normalized_score_rmse":0.2198009543113406,"constant_score_baseline":0.25,"constant_mae_baseline":0.264375,"baseline_adjusted_skill":0.2975225134732876,"baseline_adjusted_score":0.2975225134732876}. Counts: {"cases": 400, "expected": 400, "succeeded": 400, "failed": 0}. Parameter-count conventions excluding lookup/embedding parameters are not model MoE active counts. No executed weight SHA is stated in this display record; exact named repository and adapter are retained.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-18 · Samples: 400

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/synthetic_relevance__nanobeir__NanoHotpotQA / test · binary_brier. Population 400 cases / 400 decisions.

Exact Cloudflare/clef; adapter clef; settings {"questions_per_call": "all-case-questions"}. Run Cloudflare__clef.

Published measured S1MB run; community execution/submission status is not assumed independent. All published diagnostics: {"binary_brier":0.03297833863231196,"accuracy":0.9026249999999996,"positive_recall":0.7369014084507042,"specificity":0.9498795180722891,"precision":0.8074074074074074,"balanced_accuracy":0.8433904632614966,"f1":0.7705449189985273,"true_positive":65.4,"false_positive":15.600000000000003,"false_negative":23.35000000000001,"true_negative":295.64999999999986,"positive_prevalence":0.22187499999999974,"majority_accuracy_baseline":0.7781250000000003,"constant_brier_baseline":0.126740234375001,"brier_skill":0.7397958209960767,"baseline_adjusted_skill":0.6867809265229932,"baseline_adjusted_score":0.6867809265229932}. Counts: {"cases": 400, "expected": 400, "succeeded": 400, "failed": 0}. Parameter-count conventions excluding lookup/embedding parameters are not model MoE active counts. No executed weight SHA is stated in this display record; exact named repository and adapter are retained.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-18 · Samples: 400

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/synthetic_relevance__nanobeir__NanoHotpotQA / test · normalized_expected_score_mae. Population 400 cases / 400 decisions.

Exact Cloudflare/clef; adapter clef; settings {"questions_per_call": "all-case-questions"}. Run Cloudflare__clef.

Published measured S1MB run; community execution/submission status is not assumed independent. All published diagnostics: {"normalized_expected_score_mae":0.16857155303470792,"normalized_score_rmse":0.2028385055547715,"constant_score_baseline":0,"constant_mae_baseline":0.225625,"baseline_adjusted_skill":0.2528684630040646,"baseline_adjusted_score":0.2528684630040646}. Counts: {"cases": 400, "expected": 400, "succeeded": 400, "failed": 0}. Parameter-count conventions excluding lookup/embedding parameters are not model MoE active counts. No executed weight SHA is stated in this display record; exact named repository and adapter are retained.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-18 · Samples: 400

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/synthetic_relevance__nanobeir__NanoMSMARCO / test · binary_brier. Population 400 cases / 400 decisions.

Exact Cloudflare/clef; adapter clef; settings {"questions_per_call": "all-case-questions"}. Run Cloudflare__clef.

Published measured S1MB run; community execution/submission status is not assumed independent. All published diagnostics: {"binary_brier":0.03755065941432374,"accuracy":0.8419999999999996,"positive_recall":0.7495274102079394,"specificity":0.8752549286199863,"precision":0.6836206896551724,"balanced_accuracy":0.8123911694139629,"f1":0.715058611361587,"true_positive":79.3,"false_positive":36.699999999999996,"false_negative":26.50000000000001,"true_negative":257.49999999999983,"positive_prevalence":0.26449999999999985,"majority_accuracy_baseline":0.7355000000000002,"constant_brier_baseline":0.11362725000000028,"brier_skill":0.6695276932749525,"baseline_adjusted_skill":0.6247823388279259,"baseline_adjusted_score":0.6247823388279259}. Counts: {"cases": 400, "expected": 400, "succeeded": 400, "failed": 0}. Parameter-count conventions excluding lookup/embedding parameters are not model MoE active counts. No executed weight SHA is stated in this display record; exact named repository and adapter are retained.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-18 · Samples: 400

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/synthetic_relevance__nanobeir__NanoMSMARCO / test · normalized_expected_score_mae. Population 400 cases / 400 decisions.

Exact Cloudflare/clef; adapter clef; settings {"questions_per_call": "all-case-questions"}. Run Cloudflare__clef.

Published measured S1MB run; community execution/submission status is not assumed independent. All published diagnostics: {"normalized_expected_score_mae":0.15517286177782807,"normalized_score_rmse":0.18944681984467074,"constant_score_baseline":0.25,"constant_mae_baseline":0.25125,"baseline_adjusted_skill":0.38239657003849514,"baseline_adjusted_score":0.38239657003849514}. Counts: {"cases": 400, "expected": 400, "succeeded": 400, "failed": 0}. Parameter-count conventions excluding lookup/embedding parameters are not model MoE active counts. No executed weight SHA is stated in this display record; exact named repository and adapter are retained.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-18 · Samples: 400

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/synthetic_relevance__nanobeir__NanoNFCorpus / test · binary_brier. Population 400 cases / 400 decisions.

Exact Cloudflare/clef; adapter clef; settings {"questions_per_call": "all-case-questions"}. Run Cloudflare__clef.

Published measured S1MB run; community execution/submission status is not assumed independent. All published diagnostics: {"binary_brier":0.05212015898274424,"accuracy":0.8304999999999993,"positive_recall":0.29242819843342027,"specificity":0.9579468150896722,"precision":0.6222222222222222,"balanced_accuracy":0.6251875067615462,"f1":0.3978685612788631,"true_positive":22.400000000000002,"false_positive":13.6,"false_negative":54.20000000000004,"true_negative":309.7999999999998,"positive_prevalence":0.1914999999999999,"majority_accuracy_baseline":0.8085000000000001,"constant_brier_baseline":0.09190274999999995,"brier_skill":0.43287704684849726,"baseline_adjusted_skill":0.25037501352309244,"baseline_adjusted_score":0.25037501352309244}. Counts: {"cases": 400, "expected": 400, "succeeded": 400, "failed": 0}. Parameter-count conventions excluding lookup/embedding parameters are not model MoE active counts. No executed weight SHA is stated in this display record; exact named repository and adapter are retained.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-18 · Samples: 400

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/synthetic_relevance__nanobeir__NanoNFCorpus / test · normalized_expected_score_mae. Population 400 cases / 400 decisions.

Exact Cloudflare/clef; adapter clef; settings {"questions_per_call": "all-case-questions"}. Run Cloudflare__clef.

Published measured S1MB run; community execution/submission status is not assumed independent. All published diagnostics: {"normalized_expected_score_mae":0.1584439049480716,"normalized_score_rmse":0.20999762100154407,"constant_score_baseline":0,"constant_mae_baseline":0.206875,"baseline_adjusted_skill":0.23410801233560563,"baseline_adjusted_score":0.23410801233560563}. Counts: {"cases": 400, "expected": 400, "succeeded": 400, "failed": 0}. Parameter-count conventions excluding lookup/embedding parameters are not model MoE active counts. No executed weight SHA is stated in this display record; exact named repository and adapter are retained.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-18 · Samples: 400

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/synthetic_relevance__nanobeir__NanoNQ / test · binary_brier. Population 400 cases / 400 decisions.

Exact Cloudflare/clef; adapter clef; settings {"questions_per_call": "all-case-questions"}. Run Cloudflare__clef.

Published measured S1MB run; community execution/submission status is not assumed independent. All published diagnostics: {"binary_brier":0.04150974895389938,"accuracy":0.8887499999999996,"positive_recall":0.6998644986449863,"specificity":0.9314837522992029,"precision":0.697972972972973,"balanced_accuracy":0.8156741254720946,"f1":0.6989174560216508,"true_positive":51.65,"false_positive":22.35,"false_negative":22.150000000000016,"true_negative":303.84999999999985,"positive_prevalence":0.1844999999999999,"majority_accuracy_baseline":0.8155000000000001,"constant_brier_baseline":0.1092222499999995,"brier_skill":0.6199515304445791,"baseline_adjusted_skill":0.6313482509441892,"baseline_adjusted_score":0.6313482509441892}. Counts: {"cases": 400, "expected": 400, "succeeded": 400, "failed": 0}. Parameter-count conventions excluding lookup/embedding parameters are not model MoE active counts. No executed weight SHA is stated in this display record; exact named repository and adapter are retained.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-18 · Samples: 400

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/synthetic_relevance__nanobeir__NanoNQ / test · normalized_expected_score_mae. Population 400 cases / 400 decisions.

Exact Cloudflare/clef; adapter clef; settings {"questions_per_call": "all-case-questions"}. Run Cloudflare__clef.

Published measured S1MB run; community execution/submission status is not assumed independent. All published diagnostics: {"normalized_expected_score_mae":0.1888176027132431,"normalized_score_rmse":0.23259872140275273,"constant_score_baseline":0,"constant_mae_baseline":0.189375,"baseline_adjusted_skill":0.0029433520092773113,"baseline_adjusted_score":0.0029433520092773113}. Counts: {"cases": 400, "expected": 400, "succeeded": 400, "failed": 0}. Parameter-count conventions excluding lookup/embedding parameters are not model MoE active counts. No executed weight SHA is stated in this display record; exact named repository and adapter are retained.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-18 · Samples: 400

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/synthetic_relevance__nanobeir__NanoSCIDOCS / test · binary_brier. Population 400 cases / 400 decisions.

Exact Cloudflare/clef; adapter clef; settings {"questions_per_call": "all-case-questions"}. Run Cloudflare__clef.

Published measured S1MB run; community execution/submission status is not assumed independent. All published diagnostics: {"binary_brier":0.05675287883924535,"accuracy":0.8387499999999996,"positive_recall":0.2611464968152867,"specificity":0.9797822706065319,"precision":0.7592592592592593,"balanced_accuracy":0.6204643837109093,"f1":0.3886255924170617,"true_positive":20.500000000000004,"false_positive":6.5,"false_negative":57.99999999999999,"true_negative":314.99999999999983,"positive_prevalence":0.19624999999999987,"majority_accuracy_baseline":0.8037500000000002,"constant_brier_baseline":0.09878593749999955,"brier_skill":0.42549637857872624,"baseline_adjusted_skill":0.24092876742181857,"baseline_adjusted_score":0.24092876742181857}. Counts: {"cases": 400, "expected": 400, "succeeded": 400, "failed": 0}. Parameter-count conventions excluding lookup/embedding parameters are not model MoE active counts. No executed weight SHA is stated in this display record; exact named repository and adapter are retained.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-18 · Samples: 400

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/synthetic_relevance__nanobeir__NanoSCIDOCS / test · normalized_expected_score_mae. Population 400 cases / 400 decisions.

Exact Cloudflare/clef; adapter clef; settings {"questions_per_call": "all-case-questions"}. Run Cloudflare__clef.

Published measured S1MB run; community execution/submission status is not assumed independent. All published diagnostics: {"normalized_expected_score_mae":0.20182069752481765,"normalized_score_rmse":0.24175385254331982,"constant_score_baseline":0,"constant_mae_baseline":0.21125,"baseline_adjusted_skill":0.04463575136180986,"baseline_adjusted_score":0.04463575136180986}. Counts: {"cases": 400, "expected": 400, "succeeded": 400, "failed": 0}. Parameter-count conventions excluding lookup/embedding parameters are not model MoE active counts. No executed weight SHA is stated in this display record; exact named repository and adapter are retained.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-18 · Samples: 400

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/synthetic_relevance__nanobeir__NanoTouche2020 / test · binary_brier. Population 392 cases / 392 decisions.

Exact Cloudflare/clef; adapter clef; settings {"questions_per_call": "all-case-questions"}. Run Cloudflare__clef.

Published measured S1MB run; community execution/submission status is not assumed independent. All published diagnostics: {"binary_brier":0.07912901485908384,"accuracy":0.7068877551020408,"positive_recall":0.8116844602609726,"specificity":0.6277976723366158,"precision":0.6220454545454543,"balanced_accuracy":0.7197410662987942,"f1":0.7043232115285638,"true_positive":136.84999999999997,"false_positive":83.15000000000006,"false_negative":31.75000000000001,"true_negative":140.25000000000003,"positive_prevalence":0.43010204081632636,"majority_accuracy_baseline":0.5698979591836737,"constant_brier_baseline":0.12359641815910102,"brier_skill":0.35977906125706627,"baseline_adjusted_skill":0.4394821325975884,"baseline_adjusted_score":0.4394821325975884}. Counts: {"cases": 392, "expected": 392, "succeeded": 392, "failed": 0}. Parameter-count conventions excluding lookup/embedding parameters are not model MoE active counts. No executed weight SHA is stated in this display record; exact named repository and adapter are retained.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-18 · Samples: 392

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/synthetic_relevance__nanobeir__NanoTouche2020 / test · normalized_expected_score_mae. Population 392 cases / 392 decisions.

Exact Cloudflare/clef; adapter clef; settings {"questions_per_call": "all-case-questions"}. Run Cloudflare__clef.

Published measured S1MB run; community execution/submission status is not assumed independent. All published diagnostics: {"normalized_expected_score_mae":0.16715380496865764,"normalized_score_rmse":0.20738063712044194,"constant_score_baseline":0.5,"constant_mae_baseline":0.2755102040816326,"baseline_adjusted_skill":0.39329359678042775,"baseline_adjusted_score":0.39329359678042775}. Counts: {"cases": 392, "expected": 392, "succeeded": 392, "failed": 0}. Parameter-count conventions excluding lookup/embedding parameters are not model MoE active counts. No executed weight SHA is stated in this display record; exact named repository and adapter are retained.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-18 · Samples: 392

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/synthetic_relevance__nanocoir / test · binary_brier. Population 800 cases / 800 decisions.

Exact Cloudflare/clef; adapter clef; settings {"questions_per_call": "all-case-questions"}. Run Cloudflare__clef.

Published measured S1MB run; community execution/submission status is not assumed independent. All published diagnostics: {"binary_brier":0.037034568836311864,"accuracy":0.9142500000000006,"positive_recall":0.6077891424075532,"specificity":0.9721355327686135,"precision":0.8046875,"balanced_accuracy":0.7899623375880833,"f1":0.6925145674585389,"true_positive":77.25,"false_positive":18.75,"false_negative":49.84999999999997,"true_negative":654.1500000000004,"positive_prevalence":0.15887500000000002,"majority_accuracy_baseline":0.841125,"constant_brier_baseline":0.09877748437499934,"brier_skill":0.6250707428859634,"baseline_adjusted_skill":0.5799246751761666,"baseline_adjusted_score":0.5799246751761666}. Counts: {"cases": 800, "expected": 800, "succeeded": 800, "failed": 0}. Parameter-count conventions excluding lookup/embedding parameters are not model MoE active counts. No executed weight SHA is stated in this display record; exact named repository and adapter are retained.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-18 · Samples: 800

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/synthetic_relevance__nanocoir / test · normalized_expected_score_mae. Population 800 cases / 800 decisions.

Exact Cloudflare/clef; adapter clef; settings {"questions_per_call": "all-case-questions"}. Run Cloudflare__clef.

Published measured S1MB run; community execution/submission status is not assumed independent. All published diagnostics: {"normalized_expected_score_mae":0.18172179955829051,"normalized_score_rmse":0.21565069158693012,"constant_score_baseline":0,"constant_mae_baseline":0.165,"baseline_adjusted_skill":-0.10134423974721529,"baseline_adjusted_score":0}. Counts: {"cases": 800, "expected": 800, "succeeded": 800, "failed": 0}. Parameter-count conventions excluding lookup/embedding parameters are not model MoE active counts. No executed weight SHA is stated in this display record; exact named repository and adapter are retained.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-18 · Samples: 800

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/temporal_nli / test · target_mass_at_prediction. Population 100 cases / 100 decisions.

Exact Cloudflare/clef; adapter clef; settings {"questions_per_call": "all-case-questions"}. Run Cloudflare__clef.

Published measured S1MB run; community execution/submission status is not assumed independent. Choice is target mass at the selected option; soft targets are not ordinary hard-label accuracy. All published diagnostics: {"target_mass_at_prediction":0.63,"uniform_accuracy_baseline":0.3333333333333329,"fixed_answer_accuracy_baseline":0.38,"baseline_adjusted_skill":0.40322580645161293,"baseline_adjusted_score":0.40322580645161293}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0}. Parameter-count conventions excluding lookup/embedding parameters are not model MoE active counts. No executed weight SHA is stated in this display record; exact named repository and adapter are retained.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-18 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/tracie / test · binary_brier. Population 100 cases / 100 decisions.

Exact Cloudflare/clef; adapter clef; settings {"questions_per_call": "all-case-questions"}. Run Cloudflare__clef.

Published measured S1MB run; community execution/submission status is not assumed independent. All published diagnostics: {"binary_brier":0.2519784780394673,"accuracy":0.59,"positive_recall":0.48,"specificity":0.7,"precision":0.6153846153846154,"balanced_accuracy":0.59,"f1":0.5393258426966292,"true_positive":24,"false_positive":15,"false_negative":26,"true_negative":35,"positive_prevalence":0.5,"majority_accuracy_baseline":0.5,"constant_brier_baseline":0.25,"brier_skill":-0.007913912157869296,"baseline_adjusted_skill":0.17999999999999994,"baseline_adjusted_score":0.17999999999999994}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0}. Parameter-count conventions excluding lookup/embedding parameters are not model MoE active counts. No executed weight SHA is stated in this display record; exact named repository and adapter are retained.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-18 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/ud_ewt / test · target_mass_at_prediction. Population 100 cases / 1509 decisions.

Exact Cloudflare/clef; adapter clef; settings {"questions_per_call": "all-case-questions"}. Run Cloudflare__clef.

Published measured S1MB run; community execution/submission status is not assumed independent. Choice is target mass at the selected option; soft targets are not ordinary hard-label accuracy. All published diagnostics: {"target_mass_at_prediction":0.5467196819085487,"uniform_accuracy_baseline":0.058823529411766766,"fixed_answer_accuracy_baseline":0.14910536779324055,"baseline_adjusted_skill":0.4672897196261682,"baseline_adjusted_score":0.4672897196261682}. Counts: {"cases": 100, "expected": 1509, "succeeded": 1509, "failed": 0}. Parameter-count conventions excluding lookup/embedding parameters are not model MoE active counts. No executed weight SHA is stated in this display record; exact named repository and adapter are retained.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-18 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/winobias / test · target_mass_at_prediction. Population 100 cases / 100 decisions.

Exact Cloudflare/clef; adapter clef; settings {"questions_per_call": "all-case-questions"}. Run Cloudflare__clef.

Published measured S1MB run; community execution/submission status is not assumed independent. Choice is target mass at the selected option; soft targets are not ordinary hard-label accuracy. All published diagnostics: {"target_mass_at_prediction":0.92,"uniform_accuracy_baseline":0.5,"fixed_answer_accuracy_baseline":0.54,"baseline_adjusted_skill":0.8260869565217392,"baseline_adjusted_score":0.8260869565217392}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0}. Parameter-count conventions excluding lookup/embedding parameters are not model MoE active counts. No executed weight SHA is stated in this display record; exact named repository and adapter are retained.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-18 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/winowhy / test · binary_brier. Population 100 cases / 100 decisions.

Exact Cloudflare/clef; adapter clef; settings {"questions_per_call": "all-case-questions"}. Run Cloudflare__clef.

Published measured S1MB run; community execution/submission status is not assumed independent. All published diagnostics: {"binary_brier":0.19229552420911286,"accuracy":0.74,"positive_recall":0.66,"specificity":0.82,"precision":0.7857142857142857,"balanced_accuracy":0.74,"f1":0.717391304347826,"true_positive":33,"false_positive":9,"false_negative":17,"true_negative":41,"positive_prevalence":0.5,"majority_accuracy_baseline":0.5,"constant_brier_baseline":0.25,"brier_skill":0.23081790316354855,"baseline_adjusted_skill":0.48,"baseline_adjusted_score":0.48}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0}. Parameter-count conventions excluding lookup/embedding parameters are not model MoE active counts. No executed weight SHA is stated in this display record; exact named repository and adapter are retained.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-18 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/wiqa / test · target_mass_at_prediction. Population 100 cases / 100 decisions.

Exact Cloudflare/clef; adapter clef; settings {"questions_per_call": "all-case-questions"}. Run Cloudflare__clef.

Published measured S1MB run; community execution/submission status is not assumed independent. Choice is target mass at the selected option; soft targets are not ordinary hard-label accuracy. All published diagnostics: {"target_mass_at_prediction":0.56,"uniform_accuracy_baseline":0.3333333333333329,"fixed_answer_accuracy_baseline":0.37,"baseline_adjusted_skill":0.3015873015873017,"baseline_adjusted_score":0.3015873015873017}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0}. Parameter-count conventions excluding lookup/embedding parameters are not model MoE active counts. No executed weight SHA is stated in this display record; exact named repository and adapter are retained.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-18 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB english-v1 · BordaScore

Published english-v1 UI scope; ClefCloudflare/clef.

Coverage 137 benchmarks. Read the source scoring specification for adjustment, clipping and scope exclusions.

Value read from the publisher’s rendered table, not recomputed from rounded components. Noul/Choice/Score and General columns are source baseline-adjusted scores. Borda is relative to the complete model cohort in this dated scope; TaskAvg is equal task weight after equal benchmark weight. Neither is native accuracy/Brier/MAE.

Checked / reported: 2026-10-18 · Samples: Not published

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB english-v1 · TaskAvg

Published english-v1 UI scope; ClefCloudflare/clef.

Coverage 137 benchmarks. Read the source scoring specification for adjustment, clipping and scope exclusions.

Value read from the publisher’s rendered table, not recomputed from rounded components. Noul/Choice/Score and General columns are source baseline-adjusted scores. Borda is relative to the complete model cohort in this dated scope; TaskAvg is equal task weight after equal benchmark weight. Neither is native accuracy/Brier/MAE.

Checked / reported: 2026-10-18 · Samples: Not published

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB english-v1 · Noul

Published english-v1 UI scope; ClefCloudflare/clef.

Coverage 137 benchmarks. Read the source scoring specification for adjustment, clipping and scope exclusions.

Value read from the publisher’s rendered table, not recomputed from rounded components. Noul/Choice/Score and General columns are source baseline-adjusted scores. Borda is relative to the complete model cohort in this dated scope; TaskAvg is equal task weight after equal benchmark weight. Neither is native accuracy/Brier/MAE.

Checked / reported: 2026-10-18 · Samples: Not published

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB english-v1 · Choice

Published english-v1 UI scope; ClefCloudflare/clef.

Coverage 137 benchmarks. Read the source scoring specification for adjustment, clipping and scope exclusions.

Value read from the publisher’s rendered table, not recomputed from rounded components. Noul/Choice/Score and General columns are source baseline-adjusted scores. Borda is relative to the complete model cohort in this dated scope; TaskAvg is equal task weight after equal benchmark weight. Neither is native accuracy/Brier/MAE.

Checked / reported: 2026-10-18 · Samples: Not published

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB english-v1 · Score

Published english-v1 UI scope; ClefCloudflare/clef.

Coverage 137 benchmarks. Read the source scoring specification for adjustment, clipping and scope exclusions.

Value read from the publisher’s rendered table, not recomputed from rounded components. Noul/Choice/Score and General columns are source baseline-adjusted scores. Borda is relative to the complete model cohort in this dated scope; TaskAvg is equal task weight after equal benchmark weight. Neither is native accuracy/Brier/MAE.

Checked / reported: 2026-10-18 · Samples: Not published

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB english-v1 · GeneralNoul

Published english-v1 UI scope; ClefCloudflare/clef.

Coverage 137 benchmarks. Read the source scoring specification for adjustment, clipping and scope exclusions.

Value read from the publisher’s rendered table, not recomputed from rounded components. Noul/Choice/Score and General columns are source baseline-adjusted scores. Borda is relative to the complete model cohort in this dated scope; TaskAvg is equal task weight after equal benchmark weight. Neither is native accuracy/Brier/MAE.

Checked / reported: 2026-10-18 · Samples: Not published

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB english-v1 · GeneralChoice

Published english-v1 UI scope; ClefCloudflare/clef.

Coverage 137 benchmarks. Read the source scoring specification for adjustment, clipping and scope exclusions.

Value read from the publisher’s rendered table, not recomputed from rounded components. Noul/Choice/Score and General columns are source baseline-adjusted scores. Borda is relative to the complete model cohort in this dated scope; TaskAvg is equal task weight after equal benchmark weight. Neither is native accuracy/Brier/MAE.

Checked / reported: 2026-10-18 · Samples: Not published

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB english-v1 · GeneralScore

Published english-v1 UI scope; ClefCloudflare/clef.

Coverage 137 benchmarks. Read the source scoring specification for adjustment, clipping and scope exclusions.

Value read from the publisher’s rendered table, not recomputed from rounded components. Noul/Choice/Score and General columns are source baseline-adjusted scores. Borda is relative to the complete model cohort in this dated scope; TaskAvg is equal task weight after equal benchmark weight. Neither is native accuracy/Brier/MAE.

Checked / reported: 2026-10-18 · Samples: Not published

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB japanese-v1 · BordaScore

Published japanese-v1 UI scope; ClefCloudflare/clef.

Coverage 27 benchmarks. Read the source scoring specification for adjustment, clipping and scope exclusions.

Value read from the publisher’s rendered table, not recomputed from rounded components. Noul/Choice/Score and General columns are source baseline-adjusted scores. Borda is relative to the complete model cohort in this dated scope; TaskAvg is equal task weight after equal benchmark weight. Neither is native accuracy/Brier/MAE.

Checked / reported: 2026-10-18 · Samples: Not published

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB japanese-v1 · TaskAvg

Published japanese-v1 UI scope; ClefCloudflare/clef.

Coverage 27 benchmarks. Read the source scoring specification for adjustment, clipping and scope exclusions.

Value read from the publisher’s rendered table, not recomputed from rounded components. Noul/Choice/Score and General columns are source baseline-adjusted scores. Borda is relative to the complete model cohort in this dated scope; TaskAvg is equal task weight after equal benchmark weight. Neither is native accuracy/Brier/MAE.

Checked / reported: 2026-10-18 · Samples: Not published

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB japanese-v1 · Noul

Published japanese-v1 UI scope; ClefCloudflare/clef.

Coverage 27 benchmarks. Read the source scoring specification for adjustment, clipping and scope exclusions.

Value read from the publisher’s rendered table, not recomputed from rounded components. Noul/Choice/Score and General columns are source baseline-adjusted scores. Borda is relative to the complete model cohort in this dated scope; TaskAvg is equal task weight after equal benchmark weight. Neither is native accuracy/Brier/MAE.

Checked / reported: 2026-10-18 · Samples: Not published

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB japanese-v1 · Choice

Published japanese-v1 UI scope; ClefCloudflare/clef.

Coverage 27 benchmarks. Read the source scoring specification for adjustment, clipping and scope exclusions.

Value read from the publisher’s rendered table, not recomputed from rounded components. Noul/Choice/Score and General columns are source baseline-adjusted scores. Borda is relative to the complete model cohort in this dated scope; TaskAvg is equal task weight after equal benchmark weight. Neither is native accuracy/Brier/MAE.

Checked / reported: 2026-10-18 · Samples: Not published

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB japanese-v1 · Score

Published japanese-v1 UI scope; ClefCloudflare/clef.

Coverage 27 benchmarks. Read the source scoring specification for adjustment, clipping and scope exclusions.

Value read from the publisher’s rendered table, not recomputed from rounded components. Noul/Choice/Score and General columns are source baseline-adjusted scores. Borda is relative to the complete model cohort in this dated scope; TaskAvg is equal task weight after equal benchmark weight. Neither is native accuracy/Brier/MAE.

Checked / reported: 2026-10-18 · Samples: Not published

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB japanese-v1 · GeneralNoul

Published japanese-v1 UI scope; ClefCloudflare/clef.

Coverage 27 benchmarks. Read the source scoring specification for adjustment, clipping and scope exclusions.

Value read from the publisher’s rendered table, not recomputed from rounded components. Noul/Choice/Score and General columns are source baseline-adjusted scores. Borda is relative to the complete model cohort in this dated scope; TaskAvg is equal task weight after equal benchmark weight. Neither is native accuracy/Brier/MAE.

Checked / reported: 2026-10-18 · Samples: Not published

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified