Examenos

BFCL formulated as typed decisions; published case exact accuracy for this board population.

Board identity vllm-sr-d3-v310; author-submitted public run; private tests evaluated by the board. v3.1.0 tag, vendor ROCm public run and evaluator CUDA private run.

Raw 0.9734; {"requests": 1694, "answered": 1694, "unsupported": 0, "errors": 0}; coverage/error rules are source-defined.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: 1694

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

SGD/SGD-X formulated as typed decisions; published macro-F1 for this board population.

Board identity vllm-sr-d3-v310; author-submitted public run; private tests evaluated by the board. v3.1.0 tag, vendor ROCm public run and evaluator CUDA private run.

Raw 0.573; {"requests": 2500, "answered": 2500, "unsupported": 0, "errors": 0}; coverage/error rules are source-defined.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: 2500

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

ContractNLI formulated as typed decisions; published macro-F1 for this board population.

Board identity vllm-sr-d3-v310; author-submitted public run; private tests evaluated by the board. v3.1.0 tag, vendor ROCm public run and evaluator CUDA private run.

Raw 0.7851; {"requests": 123, "answered": 123, "unsupported": 0, "errors": 0}; coverage/error rules are source-defined.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: 123

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

ANLI formulated as typed decisions; published macro-F1 for this board population.

Board identity vllm-sr-d3-v310; author-submitted public run; private tests evaluated by the board. v3.1.0 tag, vendor ROCm public run and evaluator CUDA private run.

Raw 0.76; {"requests": 3200, "answered": 3200, "unsupported": 0, "errors": 0}; coverage/error rules are source-defined.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: 3200

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

ToolRet formulated as typed decisions; published nDCG@10 for this board population.

Board identity vllm-sr-d3-v310; author-submitted public run; private tests evaluated by the board. v3.1.0 tag, vendor ROCm public run and evaluator CUDA private run.

Raw 0.6909; {"requests": 685, "answered": 685, "unsupported": 0, "errors": 0}; coverage/error rules are source-defined.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: 685

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

BPoMP formulated as typed decisions; published accuracy for this board population.

Board identity vllm-sr-d3-v310; author-submitted public run; private tests evaluated by the board. v3.1.0 tag, vendor ROCm public run and evaluator CUDA private run.

Raw 0.9582; {"requests": 5000, "answered": 5000, "unsupported": 0, "errors": 0}; coverage/error rules are source-defined.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: 5000

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

Humicroedit formulated as typed decisions; published accuracy for this board population.

Board identity vllm-sr-d3-v310; author-submitted public run; private tests evaluated by the board. v3.1.0 tag, vendor ROCm public run and evaluator CUDA private run.

Raw 0.637; {"requests": 2628, "answered": 2628, "unsupported": 0, "errors": 0}; coverage/error rules are source-defined.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: 2628

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

POP909-CL formulated as typed decisions; published accuracy for this board population.

Board identity vllm-sr-d3-v310; author-submitted public run; private tests evaluated by the board. v3.1.0 tag, vendor ROCm public run and evaluator CUDA private run.

Raw 0.629; {"requests": 2000, "answered": 2000, "unsupported": 0, "errors": 0}; coverage/error rules are source-defined.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: 2000

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

cfcolor formulated as typed decisions; published accuracy for this board population.

Board identity vllm-sr-d3-v310; author-submitted public run; private tests evaluated by the board. v3.1.0 tag, vendor ROCm public run and evaluator CUDA private run.

Raw 0.656; {"requests": 5000, "answered": 5000, "unsupported": 0, "errors": 0}; coverage/error rules are source-defined.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: 5000

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

MMLU formulated as typed decisions; published accuracy for this board population.

Board identity vllm-sr-d3-v310; author-submitted public run; private tests evaluated by the board. v3.1.0 tag, vendor ROCm public run and evaluator CUDA private run.

Raw 0.885; {"requests": 14033, "answered": 14033, "unsupported": 0, "errors": 0}; coverage/error rules are source-defined.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: 14033

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

GPQA Diamond formulated as typed decisions; published accuracy for this board population.

Board identity vllm-sr-d3-v310; author-submitted public run; private tests evaluated by the board. v3.1.0 tag, vendor ROCm public run and evaluator CUDA private run.

Raw 0.5663; {"requests": 196, "answered": 196, "unsupported": 0, "errors": 0}; coverage/error rules are source-defined.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: 196

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

ARC-Easy formulated as typed decisions; published accuracy for this board population.

Board identity vllm-sr-d3-v310; author-submitted public run; private tests evaluated by the board. v3.1.0 tag, vendor ROCm public run and evaluator CUDA private run.

Raw 0.9933; {"requests": 2376, "answered": 2376, "unsupported": 0, "errors": 0}; coverage/error rules are source-defined.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: 2376

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

ARC-Challenge formulated as typed decisions; published accuracy for this board population.

Board identity vllm-sr-d3-v310; author-submitted public run; private tests evaluated by the board. v3.1.0 tag, vendor ROCm public run and evaluator CUDA private run.

Raw 0.9744; {"requests": 1172, "answered": 1172, "unsupported": 0, "errors": 0}; coverage/error rules are source-defined.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: 1172

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

WinoGrande formulated as typed decisions; published accuracy for this board population.

Board identity vllm-sr-d3-v310; author-submitted public run; private tests evaluated by the board. v3.1.0 tag, vendor ROCm public run and evaluator CUDA private run.

Raw 0.9542; {"requests": 1267, "answered": 1267, "unsupported": 0, "errors": 0}; coverage/error rules are source-defined.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: 1267

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

HellaSwag formulated as typed decisions; published accuracy for this board population.

Board identity vllm-sr-d3-v310; author-submitted public run; private tests evaluated by the board. v3.1.0 tag, vendor ROCm public run and evaluator CUDA private run.

Raw 0.9821; {"requests": 10042, "answered": 10042, "unsupported": 0, "errors": 0}; coverage/error rules are source-defined.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: 10042

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

API-Bank formulated as typed decisions; published accuracy for this board population.

Board identity vllm-sr-d3-v310; author-submitted public run; private tests evaluated by the board. v3.1.0 tag, vendor ROCm public run and evaluator CUDA private run.

Raw 0.8642; {"requests": 508, "answered": 508, "unsupported": 0, "errors": 0}; coverage/error rules are source-defined.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: 508

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

GSM8K formulated as typed decisions; published accuracy for this board population.

Board identity vllm-sr-d3-v310; author-submitted public run; private tests evaluated by the board. v3.1.0 tag, vendor ROCm public run and evaluator CUDA private run.

Raw 0.8518; {"requests": 2638, "answered": 2638, "unsupported": 0, "errors": 0}; coverage/error rules are source-defined.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: 2638

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

ChessBench formulated as typed decisions; published accuracy for this board population.

Board identity vllm-sr-d3-v310; author-submitted public run; private tests evaluated by the board. v3.1.0 tag, vendor ROCm public run and evaluator CUDA private run.

Raw 0.1758; {"requests": 5000, "answered": 5000, "unsupported": 0, "errors": 0}; coverage/error rules are source-defined.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: 5000

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

MuSR formulated as typed decisions; published accuracy for this board population.

Board identity vllm-sr-d3-v310; author-submitted public run; private tests evaluated by the board. v3.1.0 tag, vendor ROCm public run and evaluator CUDA private run.

Raw 0.6928; {"requests": 752, "answered": 752, "unsupported": 0, "errors": 0}; coverage/error rules are source-defined.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: 752

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

SATA-Bench formulated as typed decisions; published case exact accuracy for this board population.

Board identity vllm-sr-d3-v310; author-submitted public run; private tests evaluated by the board. v3.1.0 tag, vendor ROCm public run and evaluator CUDA private run.

Raw 0.3842; {"requests": 1650, "answered": 1650, "unsupported": 0, "errors": 0}; coverage/error rules are source-defined.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: 1650

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

BRIGHT formulated as typed decisions; published nDCG@10 for this board population.

Board identity vllm-sr-d3-v310; author-submitted public run; private tests evaluated by the board. v3.1.0 tag, vendor ROCm public run and evaluator CUDA private run.

Raw 0.4927; {"requests": 220, "answered": 220, "unsupported": 0, "errors": 0}; coverage/error rules are source-defined.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: 220

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

Amazon ESCI formulated as typed decisions; published macro-F1 for this board population.

Board identity vllm-sr-d3-v310; author-submitted public run; private tests evaluated by the board. v3.1.0 tag, vendor ROCm public run and evaluator CUDA private run.

Raw 0.5901; {"requests": 5000, "answered": 5000, "unsupported": 0, "errors": 0}; coverage/error rules are source-defined.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: 5000

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

ACOS formulated as typed decisions; published per-review F1 for this board population.

Board identity vllm-sr-d3-v310; author-submitted public run; private tests evaluated by the board. v3.1.0 tag, vendor ROCm public run and evaluator CUDA private run.

Raw 0.3005; {"requests": 1565, "answered": 1565, "unsupported": 0, "errors": 0}; coverage/error rules are source-defined.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: 1565

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

FinEntity formulated as typed decisions; published macro-F1 for this board population.

Board identity vllm-sr-d3-v310; author-submitted public run; private tests evaluated by the board. v3.1.0 tag, vendor ROCm public run and evaluator CUDA private run.

Raw 0.957; {"requests": 979, "answered": 979, "unsupported": 0, "errors": 0}; coverage/error rules are source-defined.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: 979

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

BANKING77 formulated as typed decisions; published macro-F1 for this board population.

Board identity vllm-sr-d3-v310; author-submitted public run; private tests evaluated by the board. v3.1.0 tag, vendor ROCm public run and evaluator CUDA private run.

Raw 0.9205; {"requests": 3080, "answered": 3080, "unsupported": 0, "errors": 0}; coverage/error rules are source-defined.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: 3080

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

iSarcasmEval formulated as typed decisions; published Sarcasm F1 · track A, English for this board population.

Board identity vllm-sr-d3-v310; author-submitted public run; private tests evaluated by the board. v3.1.0 tag, vendor ROCm public run and evaluator CUDA private run.

Raw 0.668; {"requests": 4600, "answered": 4600, "unsupported": 0, "errors": 0}; coverage/error rules are source-defined.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: 4600

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

VAST formulated as typed decisions; published macro-F1 for this board population.

Board identity vllm-sr-d3-v310; author-submitted public run; private tests evaluated by the board. v3.1.0 tag, vendor ROCm public run and evaluator CUDA private run.

Raw 0.7477; {"requests": 3006, "answered": 3006, "unsupported": 0, "errors": 0}; coverage/error rules are source-defined.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: 3006

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

NLI4CT formulated as typed decisions; published macro-F1 for this board population.

Board identity vllm-sr-d3-v310; author-submitted public run; private tests evaluated by the board. v3.1.0 tag, vendor ROCm public run and evaluator CUDA private run.

Raw 0.86; {"requests": 5500, "answered": 5500, "unsupported": 0, "errors": 0}; coverage/error rules are source-defined.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: 5500

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

CRUXEval formulated as typed decisions; published accuracy for this board population.

Board identity vllm-sr-d3-v310; author-submitted public run; private tests evaluated by the board. v3.1.0 tag, vendor ROCm public run and evaluator CUDA private run.

Raw 0.8807; {"requests": 570, "answered": 570, "unsupported": 0, "errors": 0}; coverage/error rules are source-defined.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: 570

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

CLadder formulated as typed decisions; published accuracy for this board population.

Board identity vllm-sr-d3-v310; author-submitted public run; private tests evaluated by the board. v3.1.0 tag, vendor ROCm public run and evaluator CUDA private run.

Raw 0.8124; {"requests": 5000, "answered": 5000, "unsupported": 0, "errors": 0}; coverage/error rules are source-defined.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: 5000

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

HLE formulated as typed decisions; published accuracy for this board population.

Board identity vllm-sr-d3-v310; author-submitted public run; private tests evaluated by the board. v3.1.0 tag, vendor ROCm public run and evaluator CUDA private run.

Raw 0.1397; {"requests": 501, "answered": 501, "unsupported": 0, "errors": 0}; coverage/error rules are source-defined.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: 501

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

CLINC150+OOS formulated as typed decisions; published macro-F1 for this board population.

Board identity vllm-sr-d3-v310; author-submitted public run; private tests evaluated by the board. v3.1.0 tag, vendor ROCm public run and evaluator CUDA private run.

Raw 0.9693; {"requests": 5500, "answered": 5500, "unsupported": 0, "errors": 0}; coverage/error rules are source-defined.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: 5500

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

Habermas Machine formulated as typed decisions; published accuracy for this board population.

Board identity vllm-sr-d3-v310; author-submitted public run; private tests evaluated by the board. v3.1.0 tag, vendor ROCm public run and evaluator CUDA private run.

Raw 0.4165; {"requests": 1676, "answered": 1676, "unsupported": 0, "errors": 0}; coverage/error rules are source-defined.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: 1676

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

PhishNChips phishing decisions formulated as typed decisions; published accuracy for this board population.

Board identity vllm-sr-d3-v310; author-submitted public run; private tests evaluated by the board. v3.1.0 tag, vendor ROCm public run and evaluator CUDA private run.

Raw 0.739; {"requests": 2000, "answered": 2000, "unsupported": 0, "errors": 0}; coverage/error rules are source-defined.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: 2000

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

MMLU-Pro formulated as typed decisions; published accuracy for this board population.

Board identity vllm-sr-d3-v310; author-submitted public run; private tests evaluated by the board. v3.1.0 tag, vendor ROCm public run and evaluator CUDA private run.

Raw 0.7304; {"requests": 12032, "answered": 12032, "unsupported": 0, "errors": 0}; coverage/error rules are source-defined.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: 12032

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

BBH formulated as typed decisions; published accuracy for this board population.

Board identity vllm-sr-d3-v310; author-submitted public run; private tests evaluated by the board. v3.1.0 tag, vendor ROCm public run and evaluator CUDA private run.

Raw 0.8565; {"requests": 5507, "answered": 5507, "unsupported": 0, "errors": 0}; coverage/error rules are source-defined.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: 5507

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

RAGTruth formulated as typed decisions; published F1 on hallucinated class for this board population.

Board identity vllm-sr-d3-v310; author-submitted public run; private tests evaluated by the board. v3.1.0 tag, vendor ROCm public run and evaluator CUDA private run.

Raw 0.8212; {"requests": 2700, "answered": 2700, "unsupported": 0, "errors": 0}; coverage/error rules are source-defined.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: 2700

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

RouterBench formulated as typed decisions; published selected quality (quality objective) for this board population.

Board identity vllm-sr-d3-v310; author-submitted public run; private tests evaluated by the board. v3.1.0 tag, vendor ROCm public run and evaluator CUDA private run.

Raw 0.7981; {"requests": 10000, "answered": 10000, "unsupported": 0, "errors": 0}; coverage/error rules are source-defined.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: 10000

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

HoVer formulated as typed decisions; published accuracy for this board population.

Board identity vllm-sr-d3-v310; author-submitted public run; private tests evaluated by the board. v3.1.0 tag, vendor ROCm public run and evaluator CUDA private run.

Raw 0.8355; {"requests": 4000, "answered": 4000, "unsupported": 0, "errors": 0}; coverage/error rules are source-defined.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: 4000

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

When2Call formulated as typed decisions; published accuracy for this board population.

Board identity vllm-sr-d3-v310; author-submitted public run; private tests evaluated by the board. v3.1.0 tag, vendor ROCm public run and evaluator CUDA private run.

Raw 0.8636; {"requests": 3652, "answered": 3652, "unsupported": 0, "errors": 0}; coverage/error rules are source-defined.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: 3652

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

New Yorker formulated as typed decisions; published accuracy for this board population.

Board identity vllm-sr-d3-v310; author-submitted public run; private tests evaluated by the board. v3.1.0 tag, vendor ROCm public run and evaluator CUDA private run.

Raw 0.8201; {"requests": 528, "answered": 528, "unsupported": 0, "errors": 0}; coverage/error rules are source-defined.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: 528

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

Home appliance simulator formulated as typed decisions; published case exact accuracy for this board population.

Board identity vllm-sr-d3-v310; author-submitted public run; private tests evaluated by the board. v3.1.0 tag, vendor ROCm public run and evaluator CUDA private run.

Raw 0.8068; {"requests": 88, "answered": 88, "unsupported": 0, "errors": 0}; coverage/error rules are source-defined.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: 88

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

Calibration on the source-defined text sample; confidence alone is not a quality ranking.

Board identity vllm-sr-d3-v310; author-submitted public run; private tests evaluated by the board. v3.1.0 tag, vendor ROCm public run and evaluator CUDA private run.

Raw acc: 0.7771; 31 benchmarks. ECE uses the publisher bins and weighting; native Brier convention stays distinct from other studies.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported. Uses roughly one sixth of requests with discrete correct answers; benchmarks receive equal weight. ToolRet, BRIGHT and RouterBench are excluded. Known training exposure is penalized.

Checked / reported: 2026-10-17 · Samples: 34406

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

Calibration on the source-defined text sample; confidence alone is not a quality ranking.

Board identity vllm-sr-d3-v310; author-submitted public run; private tests evaluated by the board. v3.1.0 tag, vendor ROCm public run and evaluator CUDA private run.

Raw brier: 0.3008; 31 benchmarks. ECE uses the publisher bins and weighting; native Brier convention stays distinct from other studies.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported. Uses roughly one sixth of requests with discrete correct answers; benchmarks receive equal weight. ToolRet, BRIGHT and RouterBench are excluded. Known training exposure is penalized.

Checked / reported: 2026-10-17 · Samples: 34406

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

Calibration on the source-defined text sample; confidence alone is not a quality ranking.

Board identity vllm-sr-d3-v310; author-submitted public run; private tests evaluated by the board. v3.1.0 tag, vendor ROCm public run and evaluator CUDA private run.

Raw conf: 0.8189; 31 benchmarks. ECE uses the publisher bins and weighting; native Brier convention stays distinct from other studies.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported. Uses roughly one sixth of requests with discrete correct answers; benchmarks receive equal weight. ToolRet, BRIGHT and RouterBench are excluded. Known training exposure is penalized.

Checked / reported: 2026-10-17 · Samples: 34406

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

Calibration on the source-defined text sample; confidence alone is not a quality ranking.

Board identity vllm-sr-d3-v310; author-submitted public run; private tests evaluated by the board. v3.1.0 tag, vendor ROCm public run and evaluator CUDA private run.

Raw ece: 0.0428; 31 benchmarks. ECE uses the publisher bins and weighting; native Brier convention stays distinct from other studies.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported. Uses roughly one sixth of requests with discrete correct answers; benchmarks receive equal weight. ToolRet, BRIGHT and RouterBench are excluded. Known training exposure is penalized.

Checked / reported: 2026-10-17 · Samples: 34406

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

Published Median latency for the board setup; hosted network and local inference are separate.

Board identity vllm-sr-d3-v310; author-submitted public run; private tests evaluated by the board. v3.1.0 tag, vendor ROCm public run and evaluator CUDA private run.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported. Local GPU/runtime can differ between submitted and evaluator runs; not isolated architecture speed.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

Published p95 latency for the board setup; hosted network and local inference are separate.

Board identity vllm-sr-d3-v310; author-submitted public run; private tests evaluated by the board. v3.1.0 tag, vendor ROCm public run and evaluator CUDA private run.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported. Local GPU/runtime can differ between submitted and evaluator runs; not isolated architecture speed.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

Text Decision Index 0.3.2 · Private new-domain score

Board identity vllm-sr-d3-v310; author-submitted public run; private tests evaluated by the board. v3.1.0 tag, vendor ROCm public run and evaluator CUDA private run.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

Text Decision Index 0.3.2 · Public score

Board identity vllm-sr-d3-v310; author-submitted public run; private tests evaluated by the board. v3.1.0 tag, vendor ROCm public run and evaluator CUDA private run.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

Text Decision Index 0.3.2 · Private same-skill score

Board identity vllm-sr-d3-v310; author-submitted public run; private tests evaluated by the board. v3.1.0 tag, vendor ROCm public run and evaluator CUDA private run.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

Text Decision Index 0.3.2 · Full score

Board identity vllm-sr-d3-v310; author-submitted public run; private tests evaluated by the board. v3.1.0 tag, vendor ROCm public run and evaluator CUDA private run.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

Typed vision decisions on BLINK; private population.

Vision board identity vllm-sr-d3-v310. Effective image budget 1.6 MP.

Vision board 0.3.2, 50% public and 50% private; public/private are distinct from the text board. Source benchmark weights and image caps apply. author cap 1,048,576 px lifted to the board cap via the documented media_kwargs (processor max 12.8 MP) Private normalized scores may be below zero; no clipping. Public sample count: 961.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-16T19:00:13Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

Typed vision decisions on BLINK; pub population.

Vision board identity vllm-sr-d3-v310. Effective image budget 1.6 MP.

Vision board 0.3.2, 50% public and 50% private; public/private are distinct from the text board. Source benchmark weights and image caps apply. author cap 1,048,576 px lifted to the board cap via the documented media_kwargs (processor max 12.8 MP) Private normalized scores may be below zero; no clipping. Public sample count: 961.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-16T19:00:13Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

Typed vision decisions on CharXiv; private population.

Vision board identity vllm-sr-d3-v310. Effective image budget 1.6 MP.

Vision board 0.3.2, 50% public and 50% private; public/private are distinct from the text board. Source benchmark weights and image caps apply. author cap 1,048,576 px lifted to the board cap via the documented media_kwargs (processor max 12.8 MP) Private normalized scores may be below zero; no clipping. Public sample count: 532.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-16T19:00:13Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

Typed vision decisions on CharXiv; pub population.

Vision board identity vllm-sr-d3-v310. Effective image budget 1.6 MP.

Vision board 0.3.2, 50% public and 50% private; public/private are distinct from the text board. Source benchmark weights and image caps apply. author cap 1,048,576 px lifted to the board cap via the documented media_kwargs (processor max 12.8 MP) Private normalized scores may be below zero; no clipping. Public sample count: 532.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-16T19:00:13Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

Typed vision decisions on CV-Bench; private population.

Vision board identity vllm-sr-d3-v310. Effective image budget 1.6 MP.

Vision board 0.3.2, 50% public and 50% private; public/private are distinct from the text board. Source benchmark weights and image caps apply. author cap 1,048,576 px lifted to the board cap via the documented media_kwargs (processor max 12.8 MP) Private normalized scores may be below zero; no clipping. Public sample count: 2038.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-16T19:00:13Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

Typed vision decisions on CV-Bench; pub population.

Vision board identity vllm-sr-d3-v310. Effective image budget 1.6 MP.

Vision board 0.3.2, 50% public and 50% private; public/private are distinct from the text board. Source benchmark weights and image caps apply. author cap 1,048,576 px lifted to the board cap via the documented media_kwargs (processor max 12.8 MP) Private normalized scores may be below zero; no clipping. Public sample count: 2038.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-16T19:00:13Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

Multimodal typed-choice vision evaluation · Full

Vision board identity vllm-sr-d3-v310. Effective image budget 1.6 MP.

Vision board 0.3.2, 50% public and 50% private; public/private are distinct from the text board. Source benchmark weights and image caps apply. author cap 1,048,576 px lifted to the board cap via the documented media_kwargs (processor max 12.8 MP)

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-16T19:00:13Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

Typed vision decisions on InfographicVQA; private population.

Vision board identity vllm-sr-d3-v310. Effective image budget 1.6 MP.

Vision board 0.3.2, 50% public and 50% private; public/private are distinct from the text board. Source benchmark weights and image caps apply. author cap 1,048,576 px lifted to the board cap via the documented media_kwargs (processor max 12.8 MP) Private normalized scores may be below zero; no clipping. Public sample count: 244.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-16T19:00:13Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

Typed vision decisions on InfographicVQA; pub population.

Vision board identity vllm-sr-d3-v310. Effective image budget 1.6 MP.

Vision board 0.3.2, 50% public and 50% private; public/private are distinct from the text board. Source benchmark weights and image caps apply. author cap 1,048,576 px lifted to the board cap via the documented media_kwargs (processor max 12.8 MP) Private normalized scores may be below zero; no clipping. Public sample count: 244.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-16T19:00:13Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

Typed vision decisions on KIE (CORD+FUNSD); private population.

Vision board identity vllm-sr-d3-v310. Effective image budget 1.6 MP.

Vision board 0.3.2, 50% public and 50% private; public/private are distinct from the text board. Source benchmark weights and image caps apply. author cap 1,048,576 px lifted to the board cap via the documented media_kwargs (processor max 12.8 MP) Private normalized scores may be below zero; no clipping. Public sample count: 2465.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-16T19:00:13Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

Typed vision decisions on KIE (CORD+FUNSD); pub population.

Vision board identity vllm-sr-d3-v310. Effective image budget 1.6 MP.

Vision board 0.3.2, 50% public and 50% private; public/private are distinct from the text board. Source benchmark weights and image caps apply. author cap 1,048,576 px lifted to the board cap via the documented media_kwargs (processor max 12.8 MP) Private normalized scores may be below zero; no clipping. Public sample count: 2465.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-16T19:00:13Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

Typed vision decisions on Mind2Web; private population.

Vision board identity vllm-sr-d3-v310. Effective image budget 1.6 MP.

Vision board 0.3.2, 50% public and 50% private; public/private are distinct from the text board. Source benchmark weights and image caps apply. author cap 1,048,576 px lifted to the board cap via the documented media_kwargs (processor max 12.8 MP) Private normalized scores may be below zero; no clipping. Public sample count: 973.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-16T19:00:13Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

Typed vision decisions on Mind2Web; pub population.

Vision board identity vllm-sr-d3-v310. Effective image budget 1.6 MP.

Vision board 0.3.2, 50% public and 50% private; public/private are distinct from the text board. Source benchmark weights and image caps apply. author cap 1,048,576 px lifted to the board cap via the documented media_kwargs (processor max 12.8 MP) Private normalized scores may be below zero; no clipping. Public sample count: 973.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-16T19:00:13Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

Typed vision decisions on MMMU-Pro vision; pub population.

Vision board identity vllm-sr-d3-v310. Effective image budget 1.6 MP.

Vision board 0.3.2, 50% public and 50% private; public/private are distinct from the text board. Source benchmark weights and image caps apply. author cap 1,048,576 px lifted to the board cap via the documented media_kwargs (processor max 12.8 MP) Private normalized scores may be below zero; no clipping. Public sample count: 1730.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-16T19:00:13Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

Typed vision decisions on Moderation (Hateful Memes); private population.

Vision board identity vllm-sr-d3-v310. Effective image budget 1.6 MP.

Vision board 0.3.2, 50% public and 50% private; public/private are distinct from the text board. Source benchmark weights and image caps apply. author cap 1,048,576 px lifted to the board cap via the documented media_kwargs (processor max 12.8 MP) Private normalized scores may be below zero; no clipping. Public sample count: 1213.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-16T19:00:13Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

Typed vision decisions on Moderation (Hateful Memes); pub population.

Vision board identity vllm-sr-d3-v310. Effective image budget 1.6 MP.

Vision board 0.3.2, 50% public and 50% private; public/private are distinct from the text board. Source benchmark weights and image caps apply. author cap 1,048,576 px lifted to the board cap via the documented media_kwargs (processor max 12.8 MP) Private normalized scores may be below zero; no clipping. Public sample count: 1213.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-16T19:00:13Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

Multimodal typed-choice vision evaluation · Private

Vision board identity vllm-sr-d3-v310. Effective image budget 1.6 MP.

Vision board 0.3.2, 50% public and 50% private; public/private are distinct from the text board. Source benchmark weights and image caps apply. author cap 1,048,576 px lifted to the board cap via the documented media_kwargs (processor max 12.8 MP)

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-16T19:00:13Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

Multimodal typed-choice vision evaluation · Public

Vision board identity vllm-sr-d3-v310. Effective image budget 1.6 MP.

Vision board 0.3.2, 50% public and 50% private; public/private are distinct from the text board. Source benchmark weights and image caps apply. author cap 1,048,576 px lifted to the board cap via the documented media_kwargs (processor max 12.8 MP)

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-16T19:00:13Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

Typed vision decisions on R-Bench-M; pub population.

Vision board identity vllm-sr-d3-v310. Effective image budget 1.6 MP.

Vision board 0.3.2, 50% public and 50% private; public/private are distinct from the text board. Source benchmark weights and image caps apply. author cap 1,048,576 px lifted to the board cap via the documented media_kwargs (processor max 12.8 MP) Private normalized scores may be below zero; no clipping. Public sample count: 665.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-16T19:00:13Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

Typed vision decisions on RealWorldQA; private population.

Vision board identity vllm-sr-d3-v310. Effective image budget 1.6 MP.

Vision board 0.3.2, 50% public and 50% private; public/private are distinct from the text board. Source benchmark weights and image caps apply. author cap 1,048,576 px lifted to the board cap via the documented media_kwargs (processor max 12.8 MP) Private normalized scores may be below zero; no clipping. Public sample count: 589.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-16T19:00:13Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

Typed vision decisions on RealWorldQA; pub population.

Vision board identity vllm-sr-d3-v310. Effective image budget 1.6 MP.

Vision board 0.3.2, 50% public and 50% private; public/private are distinct from the text board. Source benchmark weights and image caps apply. author cap 1,048,576 px lifted to the board cap via the documented media_kwargs (processor max 12.8 MP) Private normalized scores may be below zero; no clipping. Public sample count: 589.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-16T19:00:13Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

Typed vision decisions on Winoground; private population.

Vision board identity vllm-sr-d3-v310. Effective image budget 1.6 MP.

Vision board 0.3.2, 50% public and 50% private; public/private are distinct from the text board. Source benchmark weights and image caps apply. author cap 1,048,576 px lifted to the board cap via the documented media_kwargs (processor max 12.8 MP) Private normalized scores may be below zero; no clipping. Public sample count: 800.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-16T19:00:13Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

Typed vision decisions on Winoground; pub population.

Vision board identity vllm-sr-d3-v310. Effective image budget 1.6 MP.

Vision board 0.3.2, 50% public and 50% private; public/private are distinct from the text board. Source benchmark weights and image caps apply. author cap 1,048,576 px lifted to the board cap via the documented media_kwargs (processor max 12.8 MP) Private normalized scores may be below zero; no clipping. Public sample count: 800.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-16T19:00:13Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

Liquid vendor’s 112 unique evaluations grouped as Average; not an independent index.

Liquid vendor-run October 15 panel. Exact d1-ultra/Jev 1.14/d3 labels; remaining endpoint revisions and served precision unspecified.

Figure-only printed quantity. Source does not publish per-task weights, complete item populations or all inference settings. Average is the vendor’s published aggregate, never recomputed from rounded domains. Peer model exact hosted revisions can be unspecified.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-15 · Revision: Release figures 2026-10-15 · Published observations; result reuse terms not specified

Liquid vendor’s 112 unique evaluations grouped as Business and new fields; not an independent index.

Liquid vendor-run October 15 panel. Exact d1-ultra/Jev 1.14/d3 labels; remaining endpoint revisions and served precision unspecified.

Figure-only printed quantity. Source does not publish per-task weights, complete item populations or all inference settings. Average is the vendor’s published aggregate, never recomputed from rounded domains. Peer model exact hosted revisions can be unspecified.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-15 · Revision: Release figures 2026-10-15 · Published observations; result reuse terms not specified

Liquid vendor’s 112 unique evaluations grouped as Judgment and taste; not an independent index.

Liquid vendor-run October 15 panel. Exact d1-ultra/Jev 1.14/d3 labels; remaining endpoint revisions and served precision unspecified.

Figure-only printed quantity. Source does not publish per-task weights, complete item populations or all inference settings. Average is the vendor’s published aggregate, never recomputed from rounded domains. Peer model exact hosted revisions can be unspecified.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-15 · Revision: Release figures 2026-10-15 · Published observations; result reuse terms not specified

Liquid vendor’s 112 unique evaluations grouped as Knowledge and reasoning; not an independent index.

Liquid vendor-run October 15 panel. Exact d1-ultra/Jev 1.14/d3 labels; remaining endpoint revisions and served precision unspecified.

Figure-only printed quantity. Source does not publish per-task weights, complete item populations or all inference settings. Average is the vendor’s published aggregate, never recomputed from rounded domains. Peer model exact hosted revisions can be unspecified.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-15 · Revision: Release figures 2026-10-15 · Published observations; result reuse terms not specified

Liquid vendor’s 112 unique evaluations grouped as Languages; not an independent index.

Liquid vendor-run October 15 panel. Exact d1-ultra/Jev 1.14/d3 labels; remaining endpoint revisions and served precision unspecified.

Figure-only printed quantity. Source does not publish per-task weights, complete item populations or all inference settings. Average is the vendor’s published aggregate, never recomputed from rounded domains. Peer model exact hosted revisions can be unspecified.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-15 · Revision: Release figures 2026-10-15 · Published observations; result reuse terms not specified

Liquid vendor’s 112 unique evaluations grouped as Pictures; not an independent index.

Liquid vendor-run October 15 panel. Exact d1-ultra/Jev 1.14/d3 labels; remaining endpoint revisions and served precision unspecified.

Figure-only printed quantity. Source does not publish per-task weights, complete item populations or all inference settings. Average is the vendor’s published aggregate, never recomputed from rounded domains. Peer model exact hosted revisions can be unspecified.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-15 · Revision: Release figures 2026-10-15 · Published observations; result reuse terms not specified

Liquid vendor’s 112 unique evaluations grouped as Robustness; not an independent index.

Liquid vendor-run October 15 panel. Exact d1-ultra/Jev 1.14/d3 labels; remaining endpoint revisions and served precision unspecified.

Figure-only printed quantity. Source does not publish per-task weights, complete item populations or all inference settings. Average is the vendor’s published aggregate, never recomputed from rounded domains. Peer model exact hosted revisions can be unspecified.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-15 · Revision: Release figures 2026-10-15 · Published observations; result reuse terms not specified

Liquid vendor’s 112 unique evaluations grouped as Safety and policy; not an independent index.

Liquid vendor-run October 15 panel. Exact d1-ultra/Jev 1.14/d3 labels; remaining endpoint revisions and served precision unspecified.

Figure-only printed quantity. Source does not publish per-task weights, complete item populations or all inference settings. Average is the vendor’s published aggregate, never recomputed from rounded domains. Peer model exact hosted revisions can be unspecified.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-15 · Revision: Release figures 2026-10-15 · Published observations; result reuse terms not specified

Liquid vendor’s 112 unique evaluations grouped as Tools and agents; not an independent index.

Liquid vendor-run October 15 panel. Exact d1-ultra/Jev 1.14/d3 labels; remaining endpoint revisions and served precision unspecified.

Figure-only printed quantity. Source does not publish per-task weights, complete item populations or all inference settings. Average is the vendor’s published aggregate, never recomputed from rounded domains. Peer model exact hosted revisions can be unspecified.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-15 · Revision: Release figures 2026-10-15 · Published observations; result reuse terms not specified

Vendor-defined mean of absolute accuracy minus chosen confidence; not interchangeable with other studies’ ECE.

Liquid vendor figure: mean absolute accuracy−confidence on chosen option; hardware/precision unpublished.

Printed figure values manually inspected. Curve coordinates are not reconstructed. Source task weighting and binning are not fully published.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-15 · Revision: Release figure 2026-10-15 · Published observations; result reuse terms not specified

Images: Jev Decision Index vision board 0.3.2 · BLINK as reported by vLLM Semantic Router

Card vllm-sr/d3 at 225be8b1e2270becab69c5fe05b26f99cbbdd66b; table Images: Jev Decision Index vision board 0.3.2.

Published table label: BLINK.

Vendor-published observation, distinct from independent and other vendor harnesses. Card quotes the 12 October Decision Index 0.3.2 board; internal image tests use at most 1,000 rows and approximate rebuilds marked †. Perception Test is 19,140 three-option video questions; defaults 2 fps. Not a current board snapshot.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-12 · Revision: 225be8b1e2270becab69c5fe05b26f99cbbdd66b · Published observations; result reuse terms not specified

Images: Jev Decision Index vision board 0.3.2 · CharXiv † as reported by vLLM Semantic Router

Card vllm-sr/d3 at 225be8b1e2270becab69c5fe05b26f99cbbdd66b; table Images: Jev Decision Index vision board 0.3.2.

Published table label: CharXiv †.

Vendor-published observation, distinct from independent and other vendor harnesses. Card quotes the 12 October Decision Index 0.3.2 board; internal image tests use at most 1,000 rows and approximate rebuilds marked †. Perception Test is 19,140 three-option video questions; defaults 2 fps. Not a current board snapshot.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-12 · Revision: 225be8b1e2270becab69c5fe05b26f99cbbdd66b · Published observations; result reuse terms not specified

Images: Jev Decision Index vision board 0.3.2 · CV-Bench as reported by vLLM Semantic Router

Card vllm-sr/d3 at 225be8b1e2270becab69c5fe05b26f99cbbdd66b; table Images: Jev Decision Index vision board 0.3.2.

Published table label: CV-Bench.

Vendor-published observation, distinct from independent and other vendor harnesses. Card quotes the 12 October Decision Index 0.3.2 board; internal image tests use at most 1,000 rows and approximate rebuilds marked †. Perception Test is 19,140 three-option video questions; defaults 2 fps. Not a current board snapshot.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-12 · Revision: 225be8b1e2270becab69c5fe05b26f99cbbdd66b · Published observations; result reuse terms not specified

Images: Jev Decision Index vision board 0.3.2 · InfographicVQA † as reported by vLLM Semantic Router

Card vllm-sr/d3 at 225be8b1e2270becab69c5fe05b26f99cbbdd66b; table Images: Jev Decision Index vision board 0.3.2.

Published table label: InfographicVQA †.

Vendor-published observation, distinct from independent and other vendor harnesses. Card quotes the 12 October Decision Index 0.3.2 board; internal image tests use at most 1,000 rows and approximate rebuilds marked †. Perception Test is 19,140 three-option video questions; defaults 2 fps. Not a current board snapshot.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-12 · Revision: 225be8b1e2270becab69c5fe05b26f99cbbdd66b · Published observations; result reuse terms not specified

Images: Jev Decision Index vision board 0.3.2 · KIE (CORD+FUNSD) † as reported by vLLM Semantic Router

Card vllm-sr/d3 at 225be8b1e2270becab69c5fe05b26f99cbbdd66b; table Images: Jev Decision Index vision board 0.3.2.

Published table label: KIE (CORD+FUNSD) †.

Vendor-published observation, distinct from independent and other vendor harnesses. Card quotes the 12 October Decision Index 0.3.2 board; internal image tests use at most 1,000 rows and approximate rebuilds marked †. Perception Test is 19,140 three-option video questions; defaults 2 fps. Not a current board snapshot.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-12 · Revision: 225be8b1e2270becab69c5fe05b26f99cbbdd66b · Published observations; result reuse terms not specified

Images: Jev Decision Index vision board 0.3.2 · Mind2Web † as reported by vLLM Semantic Router

Card vllm-sr/d3 at 225be8b1e2270becab69c5fe05b26f99cbbdd66b; table Images: Jev Decision Index vision board 0.3.2.

Published table label: Mind2Web †.

Vendor-published observation, distinct from independent and other vendor harnesses. Card quotes the 12 October Decision Index 0.3.2 board; internal image tests use at most 1,000 rows and approximate rebuilds marked †. Perception Test is 19,140 three-option video questions; defaults 2 fps. Not a current board snapshot.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-12 · Revision: 225be8b1e2270becab69c5fe05b26f99cbbdd66b · Published observations; result reuse terms not specified

Images: Jev Decision Index vision board 0.3.2 · MMMU-Pro vision as reported by vLLM Semantic Router

Card vllm-sr/d3 at 225be8b1e2270becab69c5fe05b26f99cbbdd66b; table Images: Jev Decision Index vision board 0.3.2.

Published table label: MMMU-Pro vision.

Vendor-published observation, distinct from independent and other vendor harnesses. Card quotes the 12 October Decision Index 0.3.2 board; internal image tests use at most 1,000 rows and approximate rebuilds marked †. Perception Test is 19,140 three-option video questions; defaults 2 fps. Not a current board snapshot.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-12 · Revision: 225be8b1e2270becab69c5fe05b26f99cbbdd66b · Published observations; result reuse terms not specified

Images: Jev Decision Index vision board 0.3.2 · Moderation (Hateful Memes) as reported by vLLM Semantic Router

Card vllm-sr/d3 at 225be8b1e2270becab69c5fe05b26f99cbbdd66b; table Images: Jev Decision Index vision board 0.3.2.

Published table label: Moderation (Hateful Memes).

Vendor-published observation, distinct from independent and other vendor harnesses. Card quotes the 12 October Decision Index 0.3.2 board; internal image tests use at most 1,000 rows and approximate rebuilds marked †. Perception Test is 19,140 three-option video questions; defaults 2 fps. Not a current board snapshot.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-12 · Revision: 225be8b1e2270becab69c5fe05b26f99cbbdd66b · Published observations; result reuse terms not specified

Images: Jev Decision Index vision board 0.3.2 · Private ↑ as reported by vLLM Semantic Router

Card vllm-sr/d3 at 225be8b1e2270becab69c5fe05b26f99cbbdd66b; table Images: Jev Decision Index vision board 0.3.2.

Published table label: Private ↑.

Vendor-published observation, distinct from independent and other vendor harnesses. Card quotes the 12 October Decision Index 0.3.2 board; internal image tests use at most 1,000 rows and approximate rebuilds marked †. Perception Test is 19,140 three-option video questions; defaults 2 fps. Not a current board snapshot.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-12 · Revision: 225be8b1e2270becab69c5fe05b26f99cbbdd66b · Published observations; result reuse terms not specified

Images: Jev Decision Index vision board 0.3.2 · Public ↑ as reported by vLLM Semantic Router

Card vllm-sr/d3 at 225be8b1e2270becab69c5fe05b26f99cbbdd66b; table Images: Jev Decision Index vision board 0.3.2.

Published table label: Public ↑.

Vendor-published observation, distinct from independent and other vendor harnesses. Card quotes the 12 October Decision Index 0.3.2 board; internal image tests use at most 1,000 rows and approximate rebuilds marked †. Perception Test is 19,140 three-option video questions; defaults 2 fps. Not a current board snapshot.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-12 · Revision: 225be8b1e2270becab69c5fe05b26f99cbbdd66b · Published observations; result reuse terms not specified

Images: Jev Decision Index vision board 0.3.2 · R-Bench-M as reported by vLLM Semantic Router

Card vllm-sr/d3 at 225be8b1e2270becab69c5fe05b26f99cbbdd66b; table Images: Jev Decision Index vision board 0.3.2.

Published table label: R-Bench-M.

Vendor-published observation, distinct from independent and other vendor harnesses. Card quotes the 12 October Decision Index 0.3.2 board; internal image tests use at most 1,000 rows and approximate rebuilds marked †. Perception Test is 19,140 three-option video questions; defaults 2 fps. Not a current board snapshot.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-12 · Revision: 225be8b1e2270becab69c5fe05b26f99cbbdd66b · Published observations; result reuse terms not specified

Images: Jev Decision Index vision board 0.3.2 · RealWorldQA as reported by vLLM Semantic Router

Card vllm-sr/d3 at 225be8b1e2270becab69c5fe05b26f99cbbdd66b; table Images: Jev Decision Index vision board 0.3.2.

Published table label: RealWorldQA.

Vendor-published observation, distinct from independent and other vendor harnesses. Card quotes the 12 October Decision Index 0.3.2 board; internal image tests use at most 1,000 rows and approximate rebuilds marked †. Perception Test is 19,140 three-option video questions; defaults 2 fps. Not a current board snapshot.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-12 · Revision: 225be8b1e2270becab69c5fe05b26f99cbbdd66b · Published observations; result reuse terms not specified

Images: Jev Decision Index vision board 0.3.2 · Vision Index ↑ as reported by vLLM Semantic Router

Card vllm-sr/d3 at 225be8b1e2270becab69c5fe05b26f99cbbdd66b; table Images: Jev Decision Index vision board 0.3.2.

Published table label: Vision Index ↑.

Vendor-published observation, distinct from independent and other vendor harnesses. Card quotes the 12 October Decision Index 0.3.2 board; internal image tests use at most 1,000 rows and approximate rebuilds marked †. Perception Test is 19,140 three-option video questions; defaults 2 fps. Not a current board snapshot.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-12 · Revision: 225be8b1e2270becab69c5fe05b26f99cbbdd66b · Published observations; result reuse terms not specified

Images: Jev Decision Index vision board 0.3.2 · Winoground as reported by vLLM Semantic Router

Card vllm-sr/d3 at 225be8b1e2270becab69c5fe05b26f99cbbdd66b; table Images: Jev Decision Index vision board 0.3.2.

Published table label: Winoground.

Vendor-published observation, distinct from independent and other vendor harnesses. Card quotes the 12 October Decision Index 0.3.2 board; internal image tests use at most 1,000 rows and approximate rebuilds marked †. Perception Test is 19,140 three-option video questions; defaults 2 fps. Not a current board snapshot.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-12 · Revision: 225be8b1e2270becab69c5fe05b26f99cbbdd66b · Published observations; result reuse terms not specified

Text: Jev Decision Index 0.3.2 · Jev Decision Index ↑ as reported by vLLM Semantic Router

Card vllm-sr/d3 at 225be8b1e2270becab69c5fe05b26f99cbbdd66b; table Text: Jev Decision Index 0.3.2.

Published table label: Jev Decision Index ↑.

Vendor-published observation, distinct from independent and other vendor harnesses. Card quotes the 12 October Decision Index 0.3.2 board; internal image tests use at most 1,000 rows and approximate rebuilds marked †. Perception Test is 19,140 three-option video questions; defaults 2 fps. Not a current board snapshot.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-12 · Revision: 225be8b1e2270becab69c5fe05b26f99cbbdd66b · Published observations; result reuse terms not specified

Text: Jev Decision Index 0.3.2 · New-domain tasks ↑ as reported by vLLM Semantic Router

Card vllm-sr/d3 at 225be8b1e2270becab69c5fe05b26f99cbbdd66b; table Text: Jev Decision Index 0.3.2.

Published table label: New-domain tasks ↑.

Vendor-published observation, distinct from independent and other vendor harnesses. Card quotes the 12 October Decision Index 0.3.2 board; internal image tests use at most 1,000 rows and approximate rebuilds marked †. Perception Test is 19,140 three-option video questions; defaults 2 fps. Not a current board snapshot.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-12 · Revision: 225be8b1e2270becab69c5fe05b26f99cbbdd66b · Published observations; result reuse terms not specified

Text: Jev Decision Index 0.3.2 · Public ↑ as reported by vLLM Semantic Router

Card vllm-sr/d3 at 225be8b1e2270becab69c5fe05b26f99cbbdd66b; table Text: Jev Decision Index 0.3.2.

Published table label: Public ↑.

Vendor-published observation, distinct from independent and other vendor harnesses. Card quotes the 12 October Decision Index 0.3.2 board; internal image tests use at most 1,000 rows and approximate rebuilds marked †. Perception Test is 19,140 three-option video questions; defaults 2 fps. Not a current board snapshot.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-12 · Revision: 225be8b1e2270becab69c5fe05b26f99cbbdd66b · Published observations; result reuse terms not specified

Text: Jev Decision Index 0.3.2 · Same-skill tests ↑ as reported by vLLM Semantic Router

Card vllm-sr/d3 at 225be8b1e2270becab69c5fe05b26f99cbbdd66b; table Text: Jev Decision Index 0.3.2.

Published table label: Same-skill tests ↑.

Vendor-published observation, distinct from independent and other vendor harnesses. Card quotes the 12 October Decision Index 0.3.2 board; internal image tests use at most 1,000 rows and approximate rebuilds marked †. Perception Test is 19,140 three-option video questions; defaults 2 fps. Not a current board snapshot.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-12 · Revision: 225be8b1e2270becab69c5fe05b26f99cbbdd66b · Published observations; result reuse terms not specified

Videos: Perception Test · Abstraction as reported by vLLM Semantic Router

Card vllm-sr/d3 at 225be8b1e2270becab69c5fe05b26f99cbbdd66b; table Videos: Perception Test.

Published table label: Abstraction.

Vendor-published observation, distinct from independent and other vendor harnesses. Card quotes the 12 October Decision Index 0.3.2 board; internal image tests use at most 1,000 rows and approximate rebuilds marked †. Perception Test is 19,140 three-option video questions; defaults 2 fps. Not a current board snapshot.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-12 · Revision: 225be8b1e2270becab69c5fe05b26f99cbbdd66b · Published observations; result reuse terms not specified

Videos: Perception Test · All 19,140 questions as reported by vLLM Semantic Router

Card vllm-sr/d3 at 225be8b1e2270becab69c5fe05b26f99cbbdd66b; table Videos: Perception Test.

Published table label: All 19,140 questions.

Vendor-published observation, distinct from independent and other vendor harnesses. Card quotes the 12 October Decision Index 0.3.2 board; internal image tests use at most 1,000 rows and approximate rebuilds marked †. Perception Test is 19,140 three-option video questions; defaults 2 fps. Not a current board snapshot.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-12 · Revision: 225be8b1e2270becab69c5fe05b26f99cbbdd66b · Published observations; result reuse terms not specified

Videos: Perception Test · Memory as reported by vLLM Semantic Router

Card vllm-sr/d3 at 225be8b1e2270becab69c5fe05b26f99cbbdd66b; table Videos: Perception Test.

Published table label: Memory.

Vendor-published observation, distinct from independent and other vendor harnesses. Card quotes the 12 October Decision Index 0.3.2 board; internal image tests use at most 1,000 rows and approximate rebuilds marked †. Perception Test is 19,140 three-option video questions; defaults 2 fps. Not a current board snapshot.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-12 · Revision: 225be8b1e2270becab69c5fe05b26f99cbbdd66b · Published observations; result reuse terms not specified

Videos: Perception Test · Physics as reported by vLLM Semantic Router

Card vllm-sr/d3 at 225be8b1e2270becab69c5fe05b26f99cbbdd66b; table Videos: Perception Test.

Published table label: Physics.

Vendor-published observation, distinct from independent and other vendor harnesses. Card quotes the 12 October Decision Index 0.3.2 board; internal image tests use at most 1,000 rows and approximate rebuilds marked †. Perception Test is 19,140 three-option video questions; defaults 2 fps. Not a current board snapshot.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-12 · Revision: 225be8b1e2270becab69c5fe05b26f99cbbdd66b · Published observations; result reuse terms not specified

Videos: Perception Test · Semantics as reported by vLLM Semantic Router

Card vllm-sr/d3 at 225be8b1e2270becab69c5fe05b26f99cbbdd66b; table Videos: Perception Test.

Published table label: Semantics.

Vendor-published observation, distinct from independent and other vendor harnesses. Card quotes the 12 October Decision Index 0.3.2 board; internal image tests use at most 1,000 rows and approximate rebuilds marked †. Perception Test is 19,140 three-option video questions; defaults 2 fps. Not a current board snapshot.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-12 · Revision: 225be8b1e2270becab69c5fe05b26f99cbbdd66b · Published observations; result reuse terms not specified

Printed public-suite chance-corrected area value, as reported in the vendor card.

Vendor figure quoting the public Decision Index 0.3.2 by area; 12 October snapshot.

Figure values visually inspected; same 0.3.2 board version but older 12 October source snapshot. No transfer of unapproved Decision 2.0 peer checkpoints.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-12 · Revision: Board figure 2026-10-12 · Published observations; result reuse terms not specified

Printed public-suite chance-corrected area value, as reported in the vendor card.

Vendor figure quoting the public Decision Index 0.3.2 by area; 12 October snapshot.

Figure values visually inspected; same 0.3.2 board version but older 12 October source snapshot. No transfer of unapproved Decision 2.0 peer checkpoints.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-12 · Revision: Board figure 2026-10-12 · Published observations; result reuse terms not specified

Printed public-suite chance-corrected area value, as reported in the vendor card.

Vendor figure quoting the public Decision Index 0.3.2 by area; 12 October snapshot.

Figure values visually inspected; same 0.3.2 board version but older 12 October source snapshot. No transfer of unapproved Decision 2.0 peer checkpoints.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-12 · Revision: Board figure 2026-10-12 · Published observations; result reuse terms not specified

Printed public-suite chance-corrected area value, as reported in the vendor card.

Vendor figure quoting the public Decision Index 0.3.2 by area; 12 October snapshot.

Figure values visually inspected; same 0.3.2 board version but older 12 October source snapshot. No transfer of unapproved Decision 2.0 peer checkpoints.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-12 · Revision: Board figure 2026-10-12 · Published observations; result reuse terms not specified

Printed public-suite chance-corrected area value, as reported in the vendor card.

Vendor figure quoting the public Decision Index 0.3.2 by area; 12 October snapshot.

Figure values visually inspected; same 0.3.2 board version but older 12 October source snapshot. No transfer of unapproved Decision 2.0 peer checkpoints.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-12 · Revision: Board figure 2026-10-12 · Published observations; result reuse terms not specified

Warm single-request p50 latency; 32k-token text, 1 question via AutoModel.

AMD MI325X; warm; AutoModel; 32k-token text, 1 question; p50.

Published cell 4.23 s / 4.24 s; seconds converted to milliseconds where stated.

One AMD Instinct MI325X. Single warm request, no remote network. Input modality, image budget and question count are part of this metric. Distinct from board and hosted API timing; published card kernel/runtime settings apply.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-13 · Revision: 225be8b1e2270becab69c5fe05b26f99cbbdd66b · Published observations; result reuse terms not specified

Warm single-request p90 latency; 32k-token text, 1 question via AutoModel.

AMD MI325X; warm; AutoModel; 32k-token text, 1 question; p90.

Published cell 4.23 s / 4.24 s; seconds converted to milliseconds where stated.

One AMD Instinct MI325X. Single warm request, no remote network. Input modality, image budget and question count are part of this metric. Distinct from board and hosted API timing; published card kernel/runtime settings apply.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-13 · Revision: 225be8b1e2270becab69c5fe05b26f99cbbdd66b · Published observations; result reuse terms not specified

Warm single-request p50 latency; 32k-token text, 1 question via Engine, batching (now the default).

AMD MI325X; warm; Engine, batching (now the default); 32k-token text, 1 question; p50.

Published cell 4.22 s / 4.23 s; seconds converted to milliseconds where stated.

One AMD Instinct MI325X. Single warm request, no remote network. Input modality, image budget and question count are part of this metric. Distinct from board and hosted API timing; published card kernel/runtime settings apply.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-13 · Revision: 225be8b1e2270becab69c5fe05b26f99cbbdd66b · Published observations; result reuse terms not specified

Warm single-request p90 latency; 32k-token text, 1 question via Engine, batching (now the default).

AMD MI325X; warm; Engine, batching (now the default); 32k-token text, 1 question; p90.

Published cell 4.22 s / 4.23 s; seconds converted to milliseconds where stated.

One AMD Instinct MI325X. Single warm request, no remote network. Input modality, image budget and question count are part of this metric. Distinct from board and hosted API timing; published card kernel/runtime settings apply.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-13 · Revision: 225be8b1e2270becab69c5fe05b26f99cbbdd66b · Published observations; result reuse terms not specified

Warm single-request p50 latency; 32k-token text, 1 question via Engine, one request per forward.

AMD MI325X; warm; Engine, one request per forward; 32k-token text, 1 question; p50.

Published cell 4.21 s / 4.23 s; seconds converted to milliseconds where stated.

One AMD Instinct MI325X. Single warm request, no remote network. Input modality, image budget and question count are part of this metric. Distinct from board and hosted API timing; published card kernel/runtime settings apply.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-13 · Revision: 225be8b1e2270becab69c5fe05b26f99cbbdd66b · Published observations; result reuse terms not specified

Warm single-request p90 latency; 32k-token text, 1 question via Engine, one request per forward.

AMD MI325X; warm; Engine, one request per forward; 32k-token text, 1 question; p90.

Published cell 4.21 s / 4.23 s; seconds converted to milliseconds where stated.

One AMD Instinct MI325X. Single warm request, no remote network. Input modality, image budget and question count are part of this metric. Distinct from board and hosted API timing; published card kernel/runtime settings apply.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-13 · Revision: 225be8b1e2270becab69c5fe05b26f99cbbdd66b · Published observations; result reuse terms not specified

Warm single-request p50 latency; 32k-token text, 1 question via Package server.

AMD MI325X; warm; Package server; 32k-token text, 1 question; p50.

Published cell 4.30 s / 4.30 s; seconds converted to milliseconds where stated.

One AMD Instinct MI325X. Single warm request, no remote network. Input modality, image budget and question count are part of this metric. Distinct from board and hosted API timing; published card kernel/runtime settings apply.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-13 · Revision: 225be8b1e2270becab69c5fe05b26f99cbbdd66b · Published observations; result reuse terms not specified

Warm single-request p90 latency; 32k-token text, 1 question via Package server.

AMD MI325X; warm; Package server; 32k-token text, 1 question; p90.

Published cell 4.30 s / 4.30 s; seconds converted to milliseconds where stated.

One AMD Instinct MI325X. Single warm request, no remote network. Input modality, image budget and question count are part of this metric. Distinct from board and hosted API timing; published card kernel/runtime settings apply.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-13 · Revision: 225be8b1e2270becab69c5fe05b26f99cbbdd66b · Published observations; result reuse terms not specified

Warm single-request p50 latency; Image (1.6 MP), 1 question via AutoModel.

AMD MI325X; warm; AutoModel; Image (1.6 MP), 1 question; p50.

Published cell 290 / 292; seconds converted to milliseconds where stated.

One AMD Instinct MI325X. Single warm request, no remote network. Input modality, image budget and question count are part of this metric. Distinct from board and hosted API timing; published card kernel/runtime settings apply.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-13 · Revision: 225be8b1e2270becab69c5fe05b26f99cbbdd66b · Published observations; result reuse terms not specified

Warm single-request p90 latency; Image (1.6 MP), 1 question via AutoModel.

AMD MI325X; warm; AutoModel; Image (1.6 MP), 1 question; p90.

Published cell 290 / 292; seconds converted to milliseconds where stated.

One AMD Instinct MI325X. Single warm request, no remote network. Input modality, image budget and question count are part of this metric. Distinct from board and hosted API timing; published card kernel/runtime settings apply.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-13 · Revision: 225be8b1e2270becab69c5fe05b26f99cbbdd66b · Published observations; result reuse terms not specified

Warm single-request p50 latency; Image (1.6 MP), 1 question via Engine, batching (now the default).

AMD MI325X; warm; Engine, batching (now the default); Image (1.6 MP), 1 question; p50.

Published cell 315 / 316; seconds converted to milliseconds where stated.

One AMD Instinct MI325X. Single warm request, no remote network. Input modality, image budget and question count are part of this metric. Distinct from board and hosted API timing; published card kernel/runtime settings apply.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-13 · Revision: 225be8b1e2270becab69c5fe05b26f99cbbdd66b · Published observations; result reuse terms not specified

Warm single-request p90 latency; Image (1.6 MP), 1 question via Engine, batching (now the default).

AMD MI325X; warm; Engine, batching (now the default); Image (1.6 MP), 1 question; p90.

Published cell 315 / 316; seconds converted to milliseconds where stated.

One AMD Instinct MI325X. Single warm request, no remote network. Input modality, image budget and question count are part of this metric. Distinct from board and hosted API timing; published card kernel/runtime settings apply.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-13 · Revision: 225be8b1e2270becab69c5fe05b26f99cbbdd66b · Published observations; result reuse terms not specified

Warm single-request p50 latency; Image (1.6 MP), 1 question via Engine, one request per forward.

AMD MI325X; warm; Engine, one request per forward; Image (1.6 MP), 1 question; p50.

Published cell 312 / 314; seconds converted to milliseconds where stated.

One AMD Instinct MI325X. Single warm request, no remote network. Input modality, image budget and question count are part of this metric. Distinct from board and hosted API timing; published card kernel/runtime settings apply.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-13 · Revision: 225be8b1e2270becab69c5fe05b26f99cbbdd66b · Published observations; result reuse terms not specified

Warm single-request p90 latency; Image (1.6 MP), 1 question via Engine, one request per forward.

AMD MI325X; warm; Engine, one request per forward; Image (1.6 MP), 1 question; p90.

Published cell 312 / 314; seconds converted to milliseconds where stated.

One AMD Instinct MI325X. Single warm request, no remote network. Input modality, image budget and question count are part of this metric. Distinct from board and hosted API timing; published card kernel/runtime settings apply.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-13 · Revision: 225be8b1e2270becab69c5fe05b26f99cbbdd66b · Published observations; result reuse terms not specified

Warm single-request p50 latency; Image (1.6 MP), 1 question via Package server.

AMD MI325X; warm; Package server; Image (1.6 MP), 1 question; p50.

Published cell 295 / 297; seconds converted to milliseconds where stated.

One AMD Instinct MI325X. Single warm request, no remote network. Input modality, image budget and question count are part of this metric. Distinct from board and hosted API timing; published card kernel/runtime settings apply.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-13 · Revision: 225be8b1e2270becab69c5fe05b26f99cbbdd66b · Published observations; result reuse terms not specified

Warm single-request p90 latency; Image (1.6 MP), 1 question via Package server.

AMD MI325X; warm; Package server; Image (1.6 MP), 1 question; p90.

Published cell 295 / 297; seconds converted to milliseconds where stated.

One AMD Instinct MI325X. Single warm request, no remote network. Input modality, image budget and question count are part of this metric. Distinct from board and hosted API timing; published card kernel/runtime settings apply.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-13 · Revision: 225be8b1e2270becab69c5fe05b26f99cbbdd66b · Published observations; result reuse terms not specified

Warm single-request p50 latency; Image (1.6 MP), 8 questions via AutoModel.

AMD MI325X; warm; AutoModel; Image (1.6 MP), 8 questions; p50.

Published cell 1.95 s / 1.95 s; seconds converted to milliseconds where stated.

One AMD Instinct MI325X. Single warm request, no remote network. Input modality, image budget and question count are part of this metric. Distinct from board and hosted API timing; published card kernel/runtime settings apply.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-13 · Revision: 225be8b1e2270becab69c5fe05b26f99cbbdd66b · Published observations; result reuse terms not specified

Warm single-request p90 latency; Image (1.6 MP), 8 questions via AutoModel.

AMD MI325X; warm; AutoModel; Image (1.6 MP), 8 questions; p90.

Published cell 1.95 s / 1.95 s; seconds converted to milliseconds where stated.

One AMD Instinct MI325X. Single warm request, no remote network. Input modality, image budget and question count are part of this metric. Distinct from board and hosted API timing; published card kernel/runtime settings apply.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-13 · Revision: 225be8b1e2270becab69c5fe05b26f99cbbdd66b · Published observations; result reuse terms not specified

Warm single-request p50 latency; Image (1.6 MP), 8 questions via Engine, batching (now the default).

AMD MI325X; warm; Engine, batching (now the default); Image (1.6 MP), 8 questions; p50.

Published cell 1.56 s / 1.57 s; seconds converted to milliseconds where stated.

One AMD Instinct MI325X. Single warm request, no remote network. Input modality, image budget and question count are part of this metric. Distinct from board and hosted API timing; published card kernel/runtime settings apply.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-13 · Revision: 225be8b1e2270becab69c5fe05b26f99cbbdd66b · Published observations; result reuse terms not specified

Warm single-request p90 latency; Image (1.6 MP), 8 questions via Engine, batching (now the default).

AMD MI325X; warm; Engine, batching (now the default); Image (1.6 MP), 8 questions; p90.

Published cell 1.56 s / 1.57 s; seconds converted to milliseconds where stated.

One AMD Instinct MI325X. Single warm request, no remote network. Input modality, image budget and question count are part of this metric. Distinct from board and hosted API timing; published card kernel/runtime settings apply.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-13 · Revision: 225be8b1e2270becab69c5fe05b26f99cbbdd66b · Published observations; result reuse terms not specified

Warm single-request p50 latency; Image (1.6 MP), 8 questions via Engine, one request per forward.

AMD MI325X; warm; Engine, one request per forward; Image (1.6 MP), 8 questions; p50.

Published cell 1.87 s / 1.88 s; seconds converted to milliseconds where stated.

One AMD Instinct MI325X. Single warm request, no remote network. Input modality, image budget and question count are part of this metric. Distinct from board and hosted API timing; published card kernel/runtime settings apply.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-13 · Revision: 225be8b1e2270becab69c5fe05b26f99cbbdd66b · Published observations; result reuse terms not specified

Warm single-request p90 latency; Image (1.6 MP), 8 questions via Engine, one request per forward.

AMD MI325X; warm; Engine, one request per forward; Image (1.6 MP), 8 questions; p90.

Published cell 1.87 s / 1.88 s; seconds converted to milliseconds where stated.

One AMD Instinct MI325X. Single warm request, no remote network. Input modality, image budget and question count are part of this metric. Distinct from board and hosted API timing; published card kernel/runtime settings apply.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-13 · Revision: 225be8b1e2270becab69c5fe05b26f99cbbdd66b · Published observations; result reuse terms not specified

Warm single-request p50 latency; Image (1.6 MP), 8 questions via Package server.

AMD MI325X; warm; Package server; Image (1.6 MP), 8 questions; p50.

Published cell 1.95 s / 1.97 s; seconds converted to milliseconds where stated.

One AMD Instinct MI325X. Single warm request, no remote network. Input modality, image budget and question count are part of this metric. Distinct from board and hosted API timing; published card kernel/runtime settings apply.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-13 · Revision: 225be8b1e2270becab69c5fe05b26f99cbbdd66b · Published observations; result reuse terms not specified

Warm single-request p90 latency; Image (1.6 MP), 8 questions via Package server.

AMD MI325X; warm; Package server; Image (1.6 MP), 8 questions; p90.

Published cell 1.95 s / 1.97 s; seconds converted to milliseconds where stated.

One AMD Instinct MI325X. Single warm request, no remote network. Input modality, image budget and question count are part of this metric. Distinct from board and hosted API timing; published card kernel/runtime settings apply.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-13 · Revision: 225be8b1e2270becab69c5fe05b26f99cbbdd66b · Published observations; result reuse terms not specified

Warm single-request p50 latency; Text, 1 question via AutoModel.

AMD MI325X; warm; AutoModel; Text, 1 question; p50.

Published cell 54.4 / 54.6; seconds converted to milliseconds where stated.

One AMD Instinct MI325X. Single warm request, no remote network. Input modality, image budget and question count are part of this metric. Distinct from board and hosted API timing; published card kernel/runtime settings apply.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-13 · Revision: 225be8b1e2270becab69c5fe05b26f99cbbdd66b · Published observations; result reuse terms not specified

Warm single-request p90 latency; Text, 1 question via AutoModel.

AMD MI325X; warm; AutoModel; Text, 1 question; p90.

Published cell 54.4 / 54.6; seconds converted to milliseconds where stated.

One AMD Instinct MI325X. Single warm request, no remote network. Input modality, image budget and question count are part of this metric. Distinct from board and hosted API timing; published card kernel/runtime settings apply.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-13 · Revision: 225be8b1e2270becab69c5fe05b26f99cbbdd66b · Published observations; result reuse terms not specified

Warm single-request p50 latency; Text, 1 question via Engine, batching (now the default).

AMD MI325X; warm; Engine, batching (now the default); Text, 1 question; p50.

Published cell 57.2 / 57.9; seconds converted to milliseconds where stated.

One AMD Instinct MI325X. Single warm request, no remote network. Input modality, image budget and question count are part of this metric. Distinct from board and hosted API timing; published card kernel/runtime settings apply.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-13 · Revision: 225be8b1e2270becab69c5fe05b26f99cbbdd66b · Published observations; result reuse terms not specified

Warm single-request p90 latency; Text, 1 question via Engine, batching (now the default).

AMD MI325X; warm; Engine, batching (now the default); Text, 1 question; p90.

Published cell 57.2 / 57.9; seconds converted to milliseconds where stated.

One AMD Instinct MI325X. Single warm request, no remote network. Input modality, image budget and question count are part of this metric. Distinct from board and hosted API timing; published card kernel/runtime settings apply.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-13 · Revision: 225be8b1e2270becab69c5fe05b26f99cbbdd66b · Published observations; result reuse terms not specified

Warm single-request p50 latency; Text, 1 question via Engine, one request per forward.

AMD MI325X; warm; Engine, one request per forward; Text, 1 question; p50.

Published cell 55.3 / 55.6; seconds converted to milliseconds where stated.

One AMD Instinct MI325X. Single warm request, no remote network. Input modality, image budget and question count are part of this metric. Distinct from board and hosted API timing; published card kernel/runtime settings apply.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-13 · Revision: 225be8b1e2270becab69c5fe05b26f99cbbdd66b · Published observations; result reuse terms not specified

Warm single-request p90 latency; Text, 1 question via Engine, one request per forward.

AMD MI325X; warm; Engine, one request per forward; Text, 1 question; p90.

Published cell 55.3 / 55.6; seconds converted to milliseconds where stated.

One AMD Instinct MI325X. Single warm request, no remote network. Input modality, image budget and question count are part of this metric. Distinct from board and hosted API timing; published card kernel/runtime settings apply.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-13 · Revision: 225be8b1e2270becab69c5fe05b26f99cbbdd66b · Published observations; result reuse terms not specified

Warm single-request p50 latency; Text, 1 question via Package server.

AMD MI325X; warm; Package server; Text, 1 question; p50.

Published cell 55.0 / 55.3; seconds converted to milliseconds where stated.

One AMD Instinct MI325X. Single warm request, no remote network. Input modality, image budget and question count are part of this metric. Distinct from board and hosted API timing; published card kernel/runtime settings apply.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-13 · Revision: 225be8b1e2270becab69c5fe05b26f99cbbdd66b · Published observations; result reuse terms not specified

Warm single-request p90 latency; Text, 1 question via Package server.

AMD MI325X; warm; Package server; Text, 1 question; p90.

Published cell 55.0 / 55.3; seconds converted to milliseconds where stated.

One AMD Instinct MI325X. Single warm request, no remote network. Input modality, image budget and question count are part of this metric. Distinct from board and hosted API timing; published card kernel/runtime settings apply.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-13 · Revision: 225be8b1e2270becab69c5fe05b26f99cbbdd66b · Published observations; result reuse terms not specified

Warm single-request p50 latency; Text, 16 questions via AutoModel.

AMD MI325X; warm; AutoModel; Text, 16 questions; p50.

Published cell 572 / 574; seconds converted to milliseconds where stated.

One AMD Instinct MI325X. Single warm request, no remote network. Input modality, image budget and question count are part of this metric. Distinct from board and hosted API timing; published card kernel/runtime settings apply.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-13 · Revision: 225be8b1e2270becab69c5fe05b26f99cbbdd66b · Published observations; result reuse terms not specified

Warm single-request p90 latency; Text, 16 questions via AutoModel.

AMD MI325X; warm; AutoModel; Text, 16 questions; p90.

Published cell 572 / 574; seconds converted to milliseconds where stated.

One AMD Instinct MI325X. Single warm request, no remote network. Input modality, image budget and question count are part of this metric. Distinct from board and hosted API timing; published card kernel/runtime settings apply.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-13 · Revision: 225be8b1e2270becab69c5fe05b26f99cbbdd66b · Published observations; result reuse terms not specified

Warm single-request p50 latency; Text, 16 questions via Engine, batching (now the default).

AMD MI325X; warm; Engine, batching (now the default); Text, 16 questions; p50.

Published cell 548 / 549; seconds converted to milliseconds where stated.

One AMD Instinct MI325X. Single warm request, no remote network. Input modality, image budget and question count are part of this metric. Distinct from board and hosted API timing; published card kernel/runtime settings apply.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-13 · Revision: 225be8b1e2270becab69c5fe05b26f99cbbdd66b · Published observations; result reuse terms not specified

Warm single-request p90 latency; Text, 16 questions via Engine, batching (now the default).

AMD MI325X; warm; Engine, batching (now the default); Text, 16 questions; p90.

Published cell 548 / 549; seconds converted to milliseconds where stated.

One AMD Instinct MI325X. Single warm request, no remote network. Input modality, image budget and question count are part of this metric. Distinct from board and hosted API timing; published card kernel/runtime settings apply.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-13 · Revision: 225be8b1e2270becab69c5fe05b26f99cbbdd66b · Published observations; result reuse terms not specified

Warm single-request p50 latency; Text, 16 questions via Engine, one request per forward.

AMD MI325X; warm; Engine, one request per forward; Text, 16 questions; p50.

Published cell 548 / 549; seconds converted to milliseconds where stated.

One AMD Instinct MI325X. Single warm request, no remote network. Input modality, image budget and question count are part of this metric. Distinct from board and hosted API timing; published card kernel/runtime settings apply.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-13 · Revision: 225be8b1e2270becab69c5fe05b26f99cbbdd66b · Published observations; result reuse terms not specified

Warm single-request p90 latency; Text, 16 questions via Engine, one request per forward.

AMD MI325X; warm; Engine, one request per forward; Text, 16 questions; p90.

Published cell 548 / 549; seconds converted to milliseconds where stated.

One AMD Instinct MI325X. Single warm request, no remote network. Input modality, image budget and question count are part of this metric. Distinct from board and hosted API timing; published card kernel/runtime settings apply.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-13 · Revision: 225be8b1e2270becab69c5fe05b26f99cbbdd66b · Published observations; result reuse terms not specified

Warm single-request p50 latency; Text, 16 questions via Package server.

AMD MI325X; warm; Package server; Text, 16 questions; p50.

Published cell 571 / 572; seconds converted to milliseconds where stated.

One AMD Instinct MI325X. Single warm request, no remote network. Input modality, image budget and question count are part of this metric. Distinct from board and hosted API timing; published card kernel/runtime settings apply.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-13 · Revision: 225be8b1e2270becab69c5fe05b26f99cbbdd66b · Published observations; result reuse terms not specified

Warm single-request p90 latency; Text, 16 questions via Package server.

AMD MI325X; warm; Package server; Text, 16 questions; p90.

Published cell 571 / 572; seconds converted to milliseconds where stated.

One AMD Instinct MI325X. Single warm request, no remote network. Input modality, image budget and question count are part of this metric. Distinct from board and hosted API timing; published card kernel/runtime settings apply.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-13 · Revision: 225be8b1e2270becab69c5fe05b26f99cbbdd66b · Published observations; result reuse terms not specified

Warm single-request p50 latency; Video (10 s, 1080p), 1 question via AutoModel.

AMD MI325X; warm; AutoModel; Video (10 s, 1080p), 1 question; p50.

Published cell 500 / 518; seconds converted to milliseconds where stated.

One AMD Instinct MI325X. Single warm request, no remote network. Input modality, image budget and question count are part of this metric. Distinct from board and hosted API timing; published card kernel/runtime settings apply.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-13 · Revision: 225be8b1e2270becab69c5fe05b26f99cbbdd66b · Published observations; result reuse terms not specified

Warm single-request p90 latency; Video (10 s, 1080p), 1 question via AutoModel.

AMD MI325X; warm; AutoModel; Video (10 s, 1080p), 1 question; p90.

Published cell 500 / 518; seconds converted to milliseconds where stated.

One AMD Instinct MI325X. Single warm request, no remote network. Input modality, image budget and question count are part of this metric. Distinct from board and hosted API timing; published card kernel/runtime settings apply.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-13 · Revision: 225be8b1e2270becab69c5fe05b26f99cbbdd66b · Published observations; result reuse terms not specified

Warm single-request p50 latency; Video (10 s, 1080p), 1 question via Engine, batching (now the default).

AMD MI325X; warm; Engine, batching (now the default); Video (10 s, 1080p), 1 question; p50.

Published cell 533 / 548; seconds converted to milliseconds where stated.

One AMD Instinct MI325X. Single warm request, no remote network. Input modality, image budget and question count are part of this metric. Distinct from board and hosted API timing; published card kernel/runtime settings apply.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-13 · Revision: 225be8b1e2270becab69c5fe05b26f99cbbdd66b · Published observations; result reuse terms not specified

Warm single-request p90 latency; Video (10 s, 1080p), 1 question via Engine, batching (now the default).

AMD MI325X; warm; Engine, batching (now the default); Video (10 s, 1080p), 1 question; p90.

Published cell 533 / 548; seconds converted to milliseconds where stated.

One AMD Instinct MI325X. Single warm request, no remote network. Input modality, image budget and question count are part of this metric. Distinct from board and hosted API timing; published card kernel/runtime settings apply.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-13 · Revision: 225be8b1e2270becab69c5fe05b26f99cbbdd66b · Published observations; result reuse terms not specified

Warm single-request p50 latency; Video (10 s, 1080p), 1 question via Engine, one request per forward.

AMD MI325X; warm; Engine, one request per forward; Video (10 s, 1080p), 1 question; p50.

Published cell 551 / 579; seconds converted to milliseconds where stated.

One AMD Instinct MI325X. Single warm request, no remote network. Input modality, image budget and question count are part of this metric. Distinct from board and hosted API timing; published card kernel/runtime settings apply.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-13 · Revision: 225be8b1e2270becab69c5fe05b26f99cbbdd66b · Published observations; result reuse terms not specified

Warm single-request p90 latency; Video (10 s, 1080p), 1 question via Engine, one request per forward.

AMD MI325X; warm; Engine, one request per forward; Video (10 s, 1080p), 1 question; p90.

Published cell 551 / 579; seconds converted to milliseconds where stated.

One AMD Instinct MI325X. Single warm request, no remote network. Input modality, image budget and question count are part of this metric. Distinct from board and hosted API timing; published card kernel/runtime settings apply.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-13 · Revision: 225be8b1e2270becab69c5fe05b26f99cbbdd66b · Published observations; result reuse terms not specified

Warm single-request p50 latency; Video (10 s, 1080p), 1 question via Package server.

AMD MI325X; warm; Package server; Video (10 s, 1080p), 1 question; p50.

Published cell 509 / 523; seconds converted to milliseconds where stated.

One AMD Instinct MI325X. Single warm request, no remote network. Input modality, image budget and question count are part of this metric. Distinct from board and hosted API timing; published card kernel/runtime settings apply.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-13 · Revision: 225be8b1e2270becab69c5fe05b26f99cbbdd66b · Published observations; result reuse terms not specified

Warm single-request p90 latency; Video (10 s, 1080p), 1 question via Package server.

AMD MI325X; warm; Package server; Video (10 s, 1080p), 1 question; p90.

Published cell 509 / 523; seconds converted to milliseconds where stated.

One AMD Instinct MI325X. Single warm request, no remote network. Input modality, image budget and question count are part of this metric. Distinct from board and hosted API timing; published card kernel/runtime settings apply.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-13 · Revision: 225be8b1e2270becab69c5fe05b26f99cbbdd66b · Published observations; result reuse terms not specified

Published serving throughput on one GPU under the source load protocol.

AMD MI325X; AutoModel; Text, 1 question.

Published throughput cell 18.4. Asterisk means no tested load achieved the stated p99 target.

AMD MI325X, closed loop with up to 64 clients. AutoModel uses back-to-back in-process calls. Starred runs miss the p99 <1 second target; saturation and tail constraints differ, so best-value ranking is withheld.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-13 · Revision: 225be8b1e2270becab69c5fe05b26f99cbbdd66b · Published observations; result reuse terms not specified

Published serving throughput on one GPU under the source load protocol.

AMD MI325X; Engine, batching (now the default); Text, 1 question.

Published throughput cell 27. Asterisk means no tested load achieved the stated p99 target.

AMD MI325X, closed loop with up to 64 clients. AutoModel uses back-to-back in-process calls. Starred runs miss the p99 <1 second target; saturation and tail constraints differ, so best-value ranking is withheld.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-13 · Revision: 225be8b1e2270becab69c5fe05b26f99cbbdd66b · Published observations; result reuse terms not specified

Published serving throughput on one GPU under the source load protocol.

AMD MI325X; Engine, one request per forward; Text, 1 question.

Published throughput cell 18.1. Asterisk means no tested load achieved the stated p99 target.

AMD MI325X, closed loop with up to 64 clients. AutoModel uses back-to-back in-process calls. Starred runs miss the p99 <1 second target; saturation and tail constraints differ, so best-value ranking is withheld.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-13 · Revision: 225be8b1e2270becab69c5fe05b26f99cbbdd66b · Published observations; result reuse terms not specified

Published serving throughput on one GPU under the source load protocol.

AMD MI325X; Package server; Text, 1 question.

Published throughput cell 17.7. Asterisk means no tested load achieved the stated p99 target.

AMD MI325X, closed loop with up to 64 clients. AutoModel uses back-to-back in-process calls. Starred runs miss the p99 <1 second target; saturation and tail constraints differ, so best-value ranking is withheld.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-13 · Revision: 225be8b1e2270becab69c5fe05b26f99cbbdd66b · Published observations; result reuse terms not specified

Published serving throughput on one GPU under the source load protocol.

AMD MI325X; AutoModel; Text, 1 question, saturation.

Published throughput cell 18.4. Asterisk means no tested load achieved the stated p99 target.

AMD MI325X, closed loop with up to 64 clients. AutoModel uses back-to-back in-process calls. Starred runs miss the p99 <1 second target; saturation and tail constraints differ, so best-value ranking is withheld.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-13 · Revision: 225be8b1e2270becab69c5fe05b26f99cbbdd66b · Published observations; result reuse terms not specified

Published serving throughput on one GPU under the source load protocol.

AMD MI325X; Engine, batching (now the default); Text, 1 question, saturation.

Published throughput cell 29. Asterisk means no tested load achieved the stated p99 target.

AMD MI325X, closed loop with up to 64 clients. AutoModel uses back-to-back in-process calls. Starred runs miss the p99 <1 second target; saturation and tail constraints differ, so best-value ranking is withheld.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-13 · Revision: 225be8b1e2270becab69c5fe05b26f99cbbdd66b · Published observations; result reuse terms not specified

Published serving throughput on one GPU under the source load protocol.

AMD MI325X; Engine, one request per forward; Text, 1 question, saturation.

Published throughput cell 18.1. Asterisk means no tested load achieved the stated p99 target.

AMD MI325X, closed loop with up to 64 clients. AutoModel uses back-to-back in-process calls. Starred runs miss the p99 <1 second target; saturation and tail constraints differ, so best-value ranking is withheld.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-13 · Revision: 225be8b1e2270becab69c5fe05b26f99cbbdd66b · Published observations; result reuse terms not specified

Published serving throughput on one GPU under the source load protocol.

AMD MI325X; Package server; Text, 1 question, saturation.

Published throughput cell 17.7. Asterisk means no tested load achieved the stated p99 target.

AMD MI325X, closed loop with up to 64 clients. AutoModel uses back-to-back in-process calls. Starred runs miss the p99 <1 second target; saturation and tail constraints differ, so best-value ranking is withheld.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-13 · Revision: 225be8b1e2270becab69c5fe05b26f99cbbdd66b · Published observations; result reuse terms not specified

Published serving throughput on one GPU under the source load protocol.

AMD MI325X; AutoModel; Video (10 s, 1080p), 1 question.

Published throughput cell 2.0. Asterisk means no tested load achieved the stated p99 target.

AMD MI325X, closed loop with up to 64 clients. AutoModel uses back-to-back in-process calls. Starred runs miss the p99 <1 second target; saturation and tail constraints differ, so best-value ranking is withheld.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-13 · Revision: 225be8b1e2270becab69c5fe05b26f99cbbdd66b · Published observations; result reuse terms not specified

Published serving throughput on one GPU under the source load protocol.

AMD MI325X; Engine, batching (now the default); Video (10 s, 1080p), 1 question.

Published throughput cell 3.7. Asterisk means no tested load achieved the stated p99 target.

AMD MI325X, closed loop with up to 64 clients. AutoModel uses back-to-back in-process calls. Starred runs miss the p99 <1 second target; saturation and tail constraints differ, so best-value ranking is withheld.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-13 · Revision: 225be8b1e2270becab69c5fe05b26f99cbbdd66b · Published observations; result reuse terms not specified

Published serving throughput on one GPU under the source load protocol.

AMD MI325X; Engine, one request per forward; Video (10 s, 1080p), 1 question.

Published throughput cell 3.7. Asterisk means no tested load achieved the stated p99 target.

AMD MI325X, closed loop with up to 64 clients. AutoModel uses back-to-back in-process calls. Starred runs miss the p99 <1 second target; saturation and tail constraints differ, so best-value ranking is withheld.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-13 · Revision: 225be8b1e2270becab69c5fe05b26f99cbbdd66b · Published observations; result reuse terms not specified

Published serving throughput on one GPU under the source load protocol.

AMD MI325X; Package server; Video (10 s, 1080p), 1 question.

Published throughput cell 2.8. Asterisk means no tested load achieved the stated p99 target.

AMD MI325X, closed loop with up to 64 clients. AutoModel uses back-to-back in-process calls. Starred runs miss the p99 <1 second target; saturation and tail constraints differ, so best-value ranking is withheld.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-13 · Revision: 225be8b1e2270becab69c5fe05b26f99cbbdd66b · Published observations; result reuse terms not specified

OneJev peer figures · Public index

OneJev launch figure, peer-reported 16 October board observation.

Vendor-run Decision Index 0.3 public kit; 37 benchmarks / 140,178 requests as reported. Source quotes peers from the 16 October board. Not AA, evaluator-private Full, or S1MB. Exact execution dates, budgets and individual run variances are not published.

Vendor-run Decision Index 0.3 public kit; 37 benchmarks / 140,178 requests as reported. Source quotes peers from the 16 October board. Not AA, evaluator-private Full, or S1MB. Exact execution dates, budgets and individual run variances are not published.

Checked / reported: 2026-10-18 · Samples: Not published

Snapshot: 2026-10-17 · Revision: OneJev 1.6 launch · 2026-10-17 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/aegis2 / test · binary_brier. Population 100 cases / 148 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"binary_brier":0.1476536070252415,"accuracy":0.7972972972972973,"positive_recall":0.6901408450704225,"specificity":0.8961038961038961,"precision":0.8596491228070176,"balanced_accuracy":0.7931223705871593,"f1":0.765625,"true_positive":49,"false_positive":8,"false_negative":22,"true_negative":69,"positive_prevalence":0.4797297297297297,"majority_accuracy_baseline":0.5202702702702703,"constant_brier_baseline":0.24958911614317017,"brier_skill":0.4084132781633638,"baseline_adjusted_skill":0.5862447411743186,"baseline_adjusted_score":0.5862447411743186}. Counts: {"cases": 100, "expected": 148, "succeeded": 148, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/aqua_rat / test · target_mass_at_prediction. Population 100 cases / 100 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Choice target mass is a fraction converted to percent; soft targets are not ordinary hard-label accuracy. Diagnostics: {"target_mass_at_prediction":0.77,"uniform_accuracy_baseline":0.2005,"fixed_answer_accuracy_baseline":0.29,"baseline_adjusted_skill":0.6760563380281691,"baseline_adjusted_score":0.6760563380281691}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/arc / test · target_mass_at_prediction. Population 100 cases / 100 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Choice target mass is a fraction converted to percent; soft targets are not ordinary hard-label accuracy. Diagnostics: {"target_mass_at_prediction":0.98,"uniform_accuracy_baseline":0.2533333333333333,"fixed_answer_accuracy_baseline":0.32,"baseline_adjusted_skill":0.9705882352941176,"baseline_adjusted_score":0.9705882352941176}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/arct / test · target_mass_at_prediction. Population 100 cases / 100 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Choice target mass is a fraction converted to percent; soft targets are not ordinary hard-label accuracy. Diagnostics: {"target_mass_at_prediction":0.94,"uniform_accuracy_baseline":0.5,"fixed_answer_accuracy_baseline":0.53,"baseline_adjusted_skill":0.8723404255319148,"baseline_adjusted_score":0.8723404255319148}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/argument_quality / test · target_mass_at_prediction. Population 100 cases / 100 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Choice target mass is a fraction converted to percent; soft targets are not ordinary hard-label accuracy. Diagnostics: {"target_mass_at_prediction":0.98,"uniform_accuracy_baseline":0.33333333333333326,"fixed_answer_accuracy_baseline":0.51,"baseline_adjusted_skill":0.9591836734693877,"baseline_adjusted_score":0.9591836734693877}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/argument_quality / test · binary_brier. Population 100 cases / 100 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"binary_brier":0.05832595704779691,"accuracy":0.6791254982799999,"positive_recall":0.7995983497571728,"specificity":0.4858392412772035,"precision":0.7138836444492753,"balanced_accuracy":0.6427187955171881,"f1":0.7543138090910297,"true_positive":49.257971467,"false_positive":19.742028533,"false_negative":12.345421639,"true_negative":18.654578361000002,"positive_prevalence":0.61603393106,"majority_accuracy_baseline":0.61603393106,"constant_brier_baseline":0.07908707655202402,"brier_skill":0.2625096338030689,"baseline_adjusted_skill":0.28543759103437627,"baseline_adjusted_score":0.28543759103437627}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/babi_nli / test · binary_brier. Population 100 cases / 100 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"binary_brier":0.018021215117890805,"accuracy":0.98,"positive_recall":0.98,"specificity":0.98,"precision":0.98,"balanced_accuracy":0.98,"f1":0.98,"true_positive":49,"false_positive":1,"false_negative":1,"true_negative":49,"positive_prevalence":0.5,"majority_accuracy_baseline":0.5,"constant_brier_baseline":0.25,"brier_skill":0.9279151395284367,"baseline_adjusted_skill":0.96,"baseline_adjusted_score":0.96}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/banking77 / test · target_mass_at_prediction. Population 100 cases / 100 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Choice target mass is a fraction converted to percent; soft targets are not ordinary hard-label accuracy. Diagnostics: {"target_mass_at_prediction":0.91,"uniform_accuracy_baseline":0.012987012987012986,"fixed_answer_accuracy_baseline":0.05,"baseline_adjusted_skill":0.9052631578947369,"baseline_adjusted_score":0.9052631578947369}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/bbq / test · target_mass_at_prediction. Population 100 cases / 100 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Choice target mass is a fraction converted to percent; soft targets are not ordinary hard-label accuracy. Diagnostics: {"target_mass_at_prediction":0.95,"uniform_accuracy_baseline":0.33333333333333326,"fixed_answer_accuracy_baseline":0.4,"baseline_adjusted_skill":0.9166666666666666,"baseline_adjusted_score":0.9166666666666666}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/boardgameqa / test · target_mass_at_prediction. Population 100 cases / 100 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Choice target mass is a fraction converted to percent; soft targets are not ordinary hard-label accuracy. Diagnostics: {"target_mass_at_prediction":0.81,"uniform_accuracy_baseline":0.33333333333333326,"fixed_answer_accuracy_baseline":0.36,"baseline_adjusted_skill":0.7031250000000001,"baseline_adjusted_score":0.7031250000000001}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/canttalk / test · binary_brier. Population 100 cases / 100 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"binary_brier":0.1524874257465515,"accuracy":0.78,"positive_recall":1,"specificity":0.56,"precision":0.6944444444444444,"balanced_accuracy":0.78,"f1":0.819672131147541,"true_positive":50,"false_positive":22,"false_negative":0,"true_negative":28,"positive_prevalence":0.5,"majority_accuracy_baseline":0.5,"constant_brier_baseline":0.25,"brier_skill":0.390050297013794,"baseline_adjusted_skill":0.56,"baseline_adjusted_score":0.56}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/civil_comments / test · binary_brier. Population 100 cases / 700 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"binary_brier":0.048096876515408754,"accuracy":0.7492313206907628,"positive_recall":0.5482524920426974,"specificity":0.842814441628554,"precision":0.6189182761129994,"balanced_accuracy":0.6955334668356257,"f1":0.5814461713874204,"true_positive":121.92690039426088,"false_positive":75.07309960573912,"false_negative":100.46497591072693,"true_negative":402.53502408927307,"positive_prevalence":0.3177026804356969,"majority_accuracy_baseline":0.6822973195643032,"constant_brier_baseline":0.08872692397768642,"brier_skill":0.4579224167908216,"baseline_adjusted_skill":0.39106693367125134,"baseline_adjusted_score":0.39106693367125134}. Counts: {"cases": 100, "expected": 700, "succeeded": 700, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/cladder / test · binary_brier. Population 100 cases / 100 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"binary_brier":0.1257640452359461,"accuracy":0.83,"positive_recall":0.74,"specificity":0.92,"precision":0.9024390243902439,"balanced_accuracy":0.8300000000000001,"f1":0.8131868131868132,"true_positive":37,"false_positive":4,"false_negative":13,"true_negative":46,"positive_prevalence":0.5,"majority_accuracy_baseline":0.5,"constant_brier_baseline":0.25,"brier_skill":0.49694381905621565,"baseline_adjusted_skill":0.6600000000000001,"baseline_adjusted_score":0.6600000000000001}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/clinc / test · target_mass_at_prediction. Population 151 cases / 151 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Choice target mass is a fraction converted to percent; soft targets are not ordinary hard-label accuracy. Diagnostics: {"target_mass_at_prediction":0.9801324503311258,"uniform_accuracy_baseline":0.006622516556291391,"fixed_answer_accuracy_baseline":0.039735099337748346,"baseline_adjusted_skill":0.9793103448275862,"baseline_adjusted_score":0.9793103448275862}. Counts: {"cases": 151, "expected": 151, "succeeded": 151, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 151

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/commonsensemoralja-noul / test · binary_brier. Population 300 cases / 300 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"binary_brier":0.05830117226291854,"accuracy":0.9233333333333333,"positive_recall":0.86,"specificity":0.9866666666666667,"precision":0.9847328244274809,"balanced_accuracy":0.9233333333333333,"f1":0.9181494661921709,"true_positive":129,"false_positive":2,"false_negative":21,"true_negative":148,"positive_prevalence":0.5,"majority_accuracy_baseline":0.5,"constant_brier_baseline":0.25,"brier_skill":0.7667953109483259,"baseline_adjusted_skill":0.8466666666666667,"baseline_adjusted_score":0.8466666666666667}. Counts: {"cases": 300, "expected": 300, "succeeded": 300, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 300

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/contract_nli / test · target_mass_at_prediction. Population 100 cases / 1700 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Choice target mass is a fraction converted to percent; soft targets are not ordinary hard-label accuracy. Diagnostics: {"target_mass_at_prediction":0.83,"uniform_accuracy_baseline":0.3333333333333333,"fixed_answer_accuracy_baseline":0.4576470588235294,"baseline_adjusted_skill":0.6865509761388285,"baseline_adjusted_score":0.6865509761388285}. Counts: {"cases": 100, "expected": 1700, "succeeded": 1700, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/corr2cause / test · target_mass_at_prediction. Population 100 cases / 100 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Choice target mass is a fraction converted to percent; soft targets are not ordinary hard-label accuracy. Diagnostics: {"target_mass_at_prediction":0.89,"uniform_accuracy_baseline":0.33333333333333326,"fixed_answer_accuracy_baseline":0.4,"baseline_adjusted_skill":0.8166666666666667,"baseline_adjusted_score":0.8166666666666667}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/creak / test · binary_brier. Population 100 cases / 100 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"binary_brier":0.036804877946375825,"accuracy":0.94,"positive_recall":0.96,"specificity":0.92,"precision":0.9230769230769231,"balanced_accuracy":0.94,"f1":0.9411764705882353,"true_positive":48,"false_positive":4,"false_negative":2,"true_negative":46,"positive_prevalence":0.5,"majority_accuracy_baseline":0.5,"constant_brier_baseline":0.25,"brier_skill":0.8527804882144967,"baseline_adjusted_skill":0.8799999999999999,"baseline_adjusted_score":0.8799999999999999}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/crows_pairs / test · target_mass_at_prediction. Population 100 cases / 100 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Choice target mass is a fraction converted to percent; soft targets are not ordinary hard-label accuracy. Diagnostics: {"target_mass_at_prediction":0.83,"uniform_accuracy_baseline":0.5,"fixed_answer_accuracy_baseline":0.52,"baseline_adjusted_skill":0.6458333333333333,"baseline_adjusted_score":0.6458333333333333}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/cvbench-depth-choice / test · target_mass_at_prediction. Population 300 cases / 300 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Choice target mass is a fraction converted to percent; soft targets are not ordinary hard-label accuracy. Diagnostics: {"target_mass_at_prediction":0.9566666666666667,"uniform_accuracy_baseline":0.5,"fixed_answer_accuracy_baseline":0.5,"baseline_adjusted_skill":0.9133333333333333,"baseline_adjusted_score":0.9133333333333333}. Counts: {"cases": 300, "expected": 300, "succeeded": 300, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 300

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/cvbench-distance-choice / test · target_mass_at_prediction. Population 300 cases / 300 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Choice target mass is a fraction converted to percent; soft targets are not ordinary hard-label accuracy. Diagnostics: {"target_mass_at_prediction":0.8933333333333333,"uniform_accuracy_baseline":0.5,"fixed_answer_accuracy_baseline":0.52,"baseline_adjusted_skill":0.7777777777777777,"baseline_adjusted_score":0.7777777777777777}. Counts: {"cases": 300, "expected": 300, "succeeded": 300, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 300

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/cvbench-relation-choice / test · target_mass_at_prediction. Population 300 cases / 300 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Choice target mass is a fraction converted to percent; soft targets are not ordinary hard-label accuracy. Diagnostics: {"target_mass_at_prediction":0.97,"uniform_accuracy_baseline":0.5,"fixed_answer_accuracy_baseline":0.5266666666666666,"baseline_adjusted_skill":0.9366197183098591,"baseline_adjusted_score":0.9366197183098591}. Counts: {"cases": 300, "expected": 300, "succeeded": 300, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 300

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/dbpedia / test · target_mass_at_prediction. Population 100 cases / 100 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Choice target mass is a fraction converted to percent; soft targets are not ordinary hard-label accuracy. Diagnostics: {"target_mass_at_prediction":0.98,"uniform_accuracy_baseline":0.07142857142857142,"fixed_answer_accuracy_baseline":0.11,"baseline_adjusted_skill":0.9775280898876404,"baseline_adjusted_score":0.9775280898876404}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/defeasible_nli / test · target_mass_at_prediction. Population 100 cases / 100 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Choice target mass is a fraction converted to percent; soft targets are not ordinary hard-label accuracy. Diagnostics: {"target_mass_at_prediction":0.95,"uniform_accuracy_baseline":0.5,"fixed_answer_accuracy_baseline":0.54,"baseline_adjusted_skill":0.8913043478260868,"baseline_adjusted_score":0.8913043478260868}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/esci / test · target_mass_at_prediction. Population 100 cases / 100 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Choice target mass is a fraction converted to percent; soft targets are not ordinary hard-label accuracy. Diagnostics: {"target_mass_at_prediction":0.5,"uniform_accuracy_baseline":0.25,"fixed_answer_accuracy_baseline":0.27,"baseline_adjusted_skill":0.3150684931506849,"baseline_adjusted_score":0.3150684931506849}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/ethics / test · target_mass_at_prediction. Population 18 cases / 18 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Choice target mass is a fraction converted to percent; soft targets are not ordinary hard-label accuracy. Diagnostics: {"target_mass_at_prediction":1,"uniform_accuracy_baseline":0.5,"fixed_answer_accuracy_baseline":0.6666666666666666,"baseline_adjusted_skill":1,"baseline_adjusted_score":1}. Counts: {"cases": 18, "expected": 18, "succeeded": 18, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 18

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/ethics / test · binary_brier. Population 82 cases / 82 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"binary_brier":0.09014326141023263,"accuracy":0.8658536585365854,"positive_recall":0.875,"specificity":0.8571428571428571,"precision":0.8536585365853658,"balanced_accuracy":0.8660714285714286,"f1":0.8641975308641975,"true_positive":35,"false_positive":6,"false_negative":5,"true_negative":36,"positive_prevalence":0.4878048780487805,"majority_accuracy_baseline":0.5121951219512195,"constant_brier_baseline":0.2498512790005949,"brier_skill":0.6392123275461881,"baseline_adjusted_skill":0.7321428571428572,"baseline_adjusted_score":0.7321428571428572}. Counts: {"cases": 82, "expected": 82, "succeeded": 82, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 82

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/ethos / test · binary_brier. Population 100 cases / 100 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"binary_brier":0.12905346006522198,"accuracy":0.81,"positive_recall":0.76,"specificity":0.86,"precision":0.8444444444444444,"balanced_accuracy":0.81,"f1":0.8,"true_positive":38,"false_positive":7,"false_negative":12,"true_negative":43,"positive_prevalence":0.5,"majority_accuracy_baseline":0.5,"constant_brier_baseline":0.25,"brier_skill":0.4837861597391121,"baseline_adjusted_skill":0.6200000000000001,"baseline_adjusted_score":0.6200000000000001}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/few_nerd / test · target_mass_at_prediction. Population 100 cases / 358 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Choice target mass is a fraction converted to percent; soft targets are not ordinary hard-label accuracy. Diagnostics: {"target_mass_at_prediction":0.8603351955307262,"uniform_accuracy_baseline":0.125,"fixed_answer_accuracy_baseline":0.2206703910614525,"baseline_adjusted_skill":0.8207885304659498,"baseline_adjusted_score":0.8207885304659498}. Counts: {"cases": 100, "expected": 358, "succeeded": 358, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/fol_nli / test · target_mass_at_prediction. Population 100 cases / 100 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Choice target mass is a fraction converted to percent; soft targets are not ordinary hard-label accuracy. Diagnostics: {"target_mass_at_prediction":0.85,"uniform_accuracy_baseline":0.33333333333333326,"fixed_answer_accuracy_baseline":0.39,"baseline_adjusted_skill":0.7540983606557377,"baseline_adjusted_score":0.7540983606557377}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/followir_core17 / test · binary_brier. Population 100 cases / 200 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"binary_brier":0.16421747226946992,"accuracy":0.785,"positive_recall":0.5454545454545454,"specificity":0.8525641025641025,"precision":0.5106382978723404,"balanced_accuracy":0.699009324009324,"f1":0.5274725274725275,"true_positive":24,"false_positive":23,"false_negative":20,"true_negative":133,"positive_prevalence":0.22,"majority_accuracy_baseline":0.78,"constant_brier_baseline":0.1716,"brier_skill":0.04302172337138743,"baseline_adjusted_skill":0.39801864801864806,"baseline_adjusted_score":0.39801864801864806}. Counts: {"cases": 100, "expected": 200, "succeeded": 200, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/followir_news21 / test · binary_brier. Population 100 cases / 200 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"binary_brier":0.1516274772141149,"accuracy":0.8,"positive_recall":0.42,"specificity":0.9266666666666666,"precision":0.65625,"balanced_accuracy":0.6733333333333333,"f1":0.5121951219512195,"true_positive":21,"false_positive":11,"false_negative":29,"true_negative":139,"positive_prevalence":0.25,"majority_accuracy_baseline":0.75,"constant_brier_baseline":0.1875,"brier_skill":0.19132012152472055,"baseline_adjusted_skill":0.3466666666666667,"baseline_adjusted_score":0.3466666666666667}. Counts: {"cases": 100, "expected": 200, "succeeded": 200, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/followir_robust04 / test · binary_brier. Population 100 cases / 200 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"binary_brier":0.04626102563140101,"accuracy":0.945,"positive_recall":0.7368421052631579,"specificity":0.9668508287292817,"precision":0.7,"balanced_accuracy":0.8518464669962198,"f1":0.717948717948718,"true_positive":14,"false_positive":6,"false_negative":5,"true_negative":175,"positive_prevalence":0.095,"majority_accuracy_baseline":0.905,"constant_brier_baseline":0.085975,"brier_skill":0.46192468006512344,"baseline_adjusted_skill":0.7036929339924396,"baseline_adjusted_score":0.7036929339924396}. Counts: {"cases": 100, "expected": 200, "succeeded": 200, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/generalization-japanese-diverse-choice / test · target_mass_at_prediction. Population 300 cases / 300 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Choice target mass is a fraction converted to percent; soft targets are not ordinary hard-label accuracy. Diagnostics: {"target_mass_at_prediction":1,"uniform_accuracy_baseline":0.3333333333333333,"fixed_answer_accuracy_baseline":0.3333333333333333,"baseline_adjusted_skill":1,"baseline_adjusted_score":1}. Counts: {"cases": 300, "expected": 300, "succeeded": 300, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 300

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/generalization-japanese-diverse-noul / test · binary_brier. Population 300 cases / 300 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"binary_brier":0.004583374821583789,"accuracy":0.9966666666666667,"positive_recall":1,"specificity":0.9933333333333333,"precision":0.9933774834437086,"balanced_accuracy":0.9966666666666666,"f1":0.9966777408637874,"true_positive":150,"false_positive":1,"false_negative":0,"true_negative":149,"positive_prevalence":0.5,"majority_accuracy_baseline":0.5,"constant_brier_baseline":0.25,"brier_skill":0.9816665007136648,"baseline_adjusted_skill":0.9933333333333332,"baseline_adjusted_score":0.9933333333333332}. Counts: {"cases": 300, "expected": 300, "succeeded": 300, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 300

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/generalization-japanese-knowledge-hard-choice / test · target_mass_at_prediction. Population 300 cases / 300 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Choice target mass is a fraction converted to percent; soft targets are not ordinary hard-label accuracy. Diagnostics: {"target_mass_at_prediction":0.9933333333333333,"uniform_accuracy_baseline":0.25,"fixed_answer_accuracy_baseline":0.25,"baseline_adjusted_skill":0.991111111111111,"baseline_adjusted_score":0.991111111111111}. Counts: {"cases": 300, "expected": 300, "succeeded": 300, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 300

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/generalization-japanese-knowledge-hard-noul / test · binary_brier. Population 300 cases / 300 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"binary_brier":0.07741558659108869,"accuracy":0.8866666666666667,"positive_recall":0.9933333333333333,"specificity":0.78,"precision":0.8186813186813187,"balanced_accuracy":0.8866666666666667,"f1":0.8975903614457831,"true_positive":149,"false_positive":33,"false_negative":1,"true_negative":117,"positive_prevalence":0.5,"majority_accuracy_baseline":0.5,"constant_brier_baseline":0.25,"brier_skill":0.6903376536356453,"baseline_adjusted_skill":0.7733333333333334,"baseline_adjusted_score":0.7733333333333334}. Counts: {"cases": 300, "expected": 300, "succeeded": 300, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 300

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/generalization-japanese-knowledge-hard-score / test · normalized_expected_score_mae. Population 300 cases / 300 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"normalized_expected_score_mae":0.05242765858841794,"normalized_score_rmse":0.1278047986872707,"constant_score_baseline":0.5,"constant_mae_baseline":0.3333333333333333,"baseline_adjusted_skill":0.8427170242347461,"baseline_adjusted_score":0.8427170242347461}. Counts: {"cases": 300, "expected": 300, "succeeded": 300, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 300

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/generalization-japanese-math-formal-hard-choice / test · target_mass_at_prediction. Population 300 cases / 300 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Choice target mass is a fraction converted to percent; soft targets are not ordinary hard-label accuracy. Diagnostics: {"target_mass_at_prediction":0.53,"uniform_accuracy_baseline":0.25,"fixed_answer_accuracy_baseline":0.25,"baseline_adjusted_skill":0.37333333333333335,"baseline_adjusted_score":0.37333333333333335}. Counts: {"cases": 300, "expected": 300, "succeeded": 300, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 300

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/generalization-japanese-math-formal-hard-noul / test · binary_brier. Population 300 cases / 300 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"binary_brier":0.23968829630283622,"accuracy":0.6166666666666667,"positive_recall":0.49333333333333335,"specificity":0.74,"precision":0.6548672566371682,"balanced_accuracy":0.6166666666666667,"f1":0.5627376425855514,"true_positive":74,"false_positive":39,"false_negative":76,"true_negative":111,"positive_prevalence":0.5,"majority_accuracy_baseline":0.5,"constant_brier_baseline":0.25,"brier_skill":0.041246814788655106,"baseline_adjusted_skill":0.2333333333333334,"baseline_adjusted_score":0.2333333333333334}. Counts: {"cases": 300, "expected": 300, "succeeded": 300, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 300

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/generalization-japanese-math-formal-hard-score / test · normalized_expected_score_mae. Population 300 cases / 300 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"normalized_expected_score_mae":0.3019253239070582,"normalized_score_rmse":0.3654868849004859,"constant_score_baseline":0.5,"constant_mae_baseline":0.3,"baseline_adjusted_skill":-0.006417746356860654,"baseline_adjusted_score":0}. Counts: {"cases": 300, "expected": 300, "succeeded": 300, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 300

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/generalization-vision-easy-choice / test · target_mass_at_prediction. Population 300 cases / 300 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Choice target mass is a fraction converted to percent; soft targets are not ordinary hard-label accuracy. Diagnostics: {"target_mass_at_prediction":0.99,"uniform_accuracy_baseline":0.25,"fixed_answer_accuracy_baseline":0.3,"baseline_adjusted_skill":0.9857142857142857,"baseline_adjusted_score":0.9857142857142857}. Counts: {"cases": 300, "expected": 300, "succeeded": 300, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 300

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/generalization-vision-easy-noul / test · binary_brier. Population 300 cases / 300 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"binary_brier":0.004303867895607935,"accuracy":0.9966666666666667,"positive_recall":0.9932885906040269,"specificity":1,"precision":1,"balanced_accuracy":0.9966442953020134,"f1":0.9966329966329966,"true_positive":148,"false_positive":0,"false_negative":1,"true_negative":151,"positive_prevalence":0.49666666666666665,"majority_accuracy_baseline":0.5033333333333334,"constant_brier_baseline":0.2499888888888889,"brier_skill":0.9827837632514905,"baseline_adjusted_skill":0.9932885906040267,"baseline_adjusted_score":0.9932885906040267}. Counts: {"cases": 300, "expected": 300, "succeeded": 300, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 300

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/generalization-vision-easy-score / test · normalized_expected_score_mae. Population 300 cases / 300 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"normalized_expected_score_mae":0.08971453922614976,"normalized_score_rmse":0.139236498886615,"constant_score_baseline":0.5,"constant_mae_baseline":0.36666666666666664,"baseline_adjusted_skill":0.7553239839286825,"baseline_adjusted_score":0.7553239839286825}. Counts: {"cases": 300, "expected": 300, "succeeded": 300, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 300

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/generalization-vision-hard-choice / test · target_mass_at_prediction. Population 300 cases / 300 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Choice target mass is a fraction converted to percent; soft targets are not ordinary hard-label accuracy. Diagnostics: {"target_mass_at_prediction":0.8766666666666667,"uniform_accuracy_baseline":0.25,"fixed_answer_accuracy_baseline":0.2866666666666667,"baseline_adjusted_skill":0.8271028037383178,"baseline_adjusted_score":0.8271028037383178}. Counts: {"cases": 300, "expected": 300, "succeeded": 300, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 300

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/generalization-vision-hard-noul / test · binary_brier. Population 300 cases / 300 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"binary_brier":0.0672817953732461,"accuracy":0.9233333333333333,"positive_recall":0.9426751592356688,"specificity":0.9020979020979021,"precision":0.9135802469135802,"balanced_accuracy":0.9223865306667854,"f1":0.9278996865203761,"true_positive":148,"false_positive":14,"false_negative":9,"true_negative":129,"positive_prevalence":0.5233333333333333,"majority_accuracy_baseline":0.5233333333333333,"constant_brier_baseline":0.24945555555555557,"brier_skill":0.7302854401321923,"baseline_adjusted_skill":0.8447730613335709,"baseline_adjusted_score":0.8447730613335709}. Counts: {"cases": 300, "expected": 300, "succeeded": 300, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 300

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/generalization-vision-hard-score / test · normalized_expected_score_mae. Population 300 cases / 300 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"normalized_expected_score_mae":0.1598203925506037,"normalized_score_rmse":0.2084431166142727,"constant_score_baseline":0.5,"constant_mae_baseline":0.3016666666666667,"baseline_adjusted_skill":0.47020864347866176,"baseline_adjusted_score":0.47020864347866176}. Counts: {"cases": 300, "expected": 300, "succeeded": 300, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 300

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/go_emotions / test · binary_brier. Population 100 cases / 2800 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"binary_brier":0.0482013795161218,"accuracy":0.8992321428571428,"positive_recall":0.6572760638825514,"specificity":0.9146777778481373,"precision":0.32965174129353236,"balanced_accuracy":0.7859769208653444,"f1":0.4390841920413506,"true_positive":110.43333333333334,"false_positive":224.56666666666666,"false_negative":57.58333333333333,"true_negative":2407.4166666666665,"positive_prevalence":0.060005952380952375,"majority_accuracy_baseline":0.9399940476190476,"constant_brier_baseline":0.03220097218679138,"brier_skill":-0.49689205768432276,"baseline_adjusted_skill":0.5719538417306889,"baseline_adjusted_score":0.5719538417306889}. Counts: {"cases": 100, "expected": 2800, "succeeded": 2800, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/gpqa-ja-main-choice / test · target_mass_at_prediction. Population 300 cases / 300 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Choice target mass is a fraction converted to percent; soft targets are not ordinary hard-label accuracy. Diagnostics: {"target_mass_at_prediction":0.5433333333333333,"uniform_accuracy_baseline":0.25,"fixed_answer_accuracy_baseline":0.25,"baseline_adjusted_skill":0.39111111111111113,"baseline_adjusted_score":0.39111111111111113}. Counts: {"cases": 300, "expected": 300, "succeeded": 300, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 300

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/gretel_pii / test · target_mass_at_prediction. Population 100 cases / 353 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Choice target mass is a fraction converted to percent; soft targets are not ordinary hard-label accuracy. Diagnostics: {"target_mass_at_prediction":0.4702549575070821,"uniform_accuracy_baseline":0.027777777777777776,"fixed_answer_accuracy_baseline":0.1274787535410765,"baseline_adjusted_skill":0.3928571428571428,"baseline_adjusted_score":0.3928571428571428}. Counts: {"cases": 100, "expected": 353, "succeeded": 353, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/gsm8k / test · target_mass_at_prediction. Population 100 cases / 100 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Choice target mass is a fraction converted to percent; soft targets are not ordinary hard-label accuracy. Diagnostics: {"target_mass_at_prediction":0.87,"uniform_accuracy_baseline":0.25,"fixed_answer_accuracy_baseline":0.3,"baseline_adjusted_skill":0.8142857142857144,"baseline_adjusted_score":0.8142857142857144}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/hans / test · binary_brier. Population 100 cases / 100 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"binary_brier":0.07551713968464886,"accuracy":0.89,"positive_recall":1,"specificity":0.78,"precision":0.819672131147541,"balanced_accuracy":0.89,"f1":0.9009009009009009,"true_positive":50,"false_positive":11,"false_negative":0,"true_negative":39,"positive_prevalence":0.5,"majority_accuracy_baseline":0.5,"constant_brier_baseline":0.25,"brier_skill":0.6979314412614046,"baseline_adjusted_skill":0.78,"baseline_adjusted_score":0.78}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/hatecheck / test · binary_brier. Population 100 cases / 100 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"binary_brier":0.003999694318826164,"accuracy":1,"positive_recall":1,"specificity":1,"precision":1,"balanced_accuracy":1,"f1":1,"true_positive":50,"false_positive":0,"false_negative":0,"true_negative":50,"positive_prevalence":0.5,"majority_accuracy_baseline":0.5,"constant_brier_baseline":0.25,"brier_skill":0.9840012227246954,"baseline_adjusted_skill":1,"baseline_adjusted_score":1}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/hh_rlhf / test · target_mass_at_prediction. Population 100 cases / 100 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Choice target mass is a fraction converted to percent; soft targets are not ordinary hard-label accuracy. Diagnostics: {"target_mass_at_prediction":0.68,"uniform_accuracy_baseline":0.5,"fixed_answer_accuracy_baseline":0.61,"baseline_adjusted_skill":0.17948717948717963,"baseline_adjusted_score":0.17948717948717963}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/hwu64 / test · target_mass_at_prediction. Population 100 cases / 100 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Choice target mass is a fraction converted to percent; soft targets are not ordinary hard-label accuracy. Diagnostics: {"target_mass_at_prediction":0.91,"uniform_accuracy_baseline":0.015625,"fixed_answer_accuracy_baseline":0.06,"baseline_adjusted_skill":0.9042553191489363,"baseline_adjusted_score":0.9042553191489363}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/impli / test · binary_brier. Population 100 cases / 100 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"binary_brier":0.04270193261158867,"accuracy":0.93,"positive_recall":0.94,"specificity":0.92,"precision":0.9215686274509803,"balanced_accuracy":0.9299999999999999,"f1":0.9306930693069307,"true_positive":47,"false_positive":4,"false_negative":3,"true_negative":46,"positive_prevalence":0.5,"majority_accuracy_baseline":0.5,"constant_brier_baseline":0.25,"brier_skill":0.8291922695536453,"baseline_adjusted_skill":0.8599999999999999,"baseline_adjusted_score":0.8599999999999999}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/jamc-qa-choice / test · target_mass_at_prediction. Population 300 cases / 300 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Choice target mass is a fraction converted to percent; soft targets are not ordinary hard-label accuracy. Diagnostics: {"target_mass_at_prediction":0.5166666666666667,"uniform_accuracy_baseline":0.25,"fixed_answer_accuracy_baseline":0.25,"baseline_adjusted_skill":0.3555555555555556,"baseline_adjusted_score":0.3555555555555556}. Counts: {"cases": 300, "expected": 300, "succeeded": 300, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 300

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/jamp-choice / test · target_mass_at_prediction. Population 300 cases / 300 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Choice target mass is a fraction converted to percent; soft targets are not ordinary hard-label accuracy. Diagnostics: {"target_mass_at_prediction":0.75,"uniform_accuracy_baseline":0.3333333333333333,"fixed_answer_accuracy_baseline":0.3333333333333333,"baseline_adjusted_skill":0.625,"baseline_adjusted_score":0.625}. Counts: {"cases": 300, "expected": 300, "succeeded": 300, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 300

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/janli-choice / test · target_mass_at_prediction. Population 285 cases / 285 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Choice target mass is a fraction converted to percent; soft targets are not ordinary hard-label accuracy. Diagnostics: {"target_mass_at_prediction":0.8105263157894737,"uniform_accuracy_baseline":0.5,"fixed_answer_accuracy_baseline":0.5263157894736842,"baseline_adjusted_skill":0.6,"baseline_adjusted_score":0.6}. Counts: {"cases": 285, "expected": 285, "succeeded": 285, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 285

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/jcola-noul / test · binary_brier. Population 299 cases / 299 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"binary_brier":0.22495898362471237,"accuracy":0.6755852842809364,"positive_recall":0.9533333333333334,"specificity":0.3959731543624161,"precision":0.6137339055793991,"balanced_accuracy":0.6746532438478747,"f1":0.7467362924281984,"true_positive":143,"false_positive":90,"false_negative":7,"true_negative":59,"positive_prevalence":0.5016722408026756,"majority_accuracy_baseline":0.5016722408026756,"constant_brier_baseline":0.24999720361069788,"brier_skill":0.10015400022224119,"baseline_adjusted_skill":0.34930648769574946,"baseline_adjusted_score":0.34930648769574946}. Counts: {"cases": 299, "expected": 299, "succeeded": 299, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 299

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/jcommonsenseqa-choice / test · target_mass_at_prediction. Population 300 cases / 300 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Choice target mass is a fraction converted to percent; soft targets are not ordinary hard-label accuracy. Diagnostics: {"target_mass_at_prediction":0.97,"uniform_accuracy_baseline":0.2,"fixed_answer_accuracy_baseline":0.2,"baseline_adjusted_skill":0.9625,"baseline_adjusted_score":0.9625}. Counts: {"cases": 300, "expected": 300, "succeeded": 300, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 300

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/jculture-mcq-noul / test · binary_brier. Population 283 cases / 283 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"binary_brier":0.22762568162692026,"accuracy":0.657243816254417,"positive_recall":0.8195488721804511,"specificity":0.5133333333333333,"precision":0.5989010989010989,"balanced_accuracy":0.6664411027568922,"f1":0.692063492063492,"true_positive":109,"false_positive":73,"false_negative":24,"true_negative":77,"positive_prevalence":0.46996466431095407,"majority_accuracy_baseline":0.5300353356890459,"constant_brier_baseline":0.24909787861004634,"brier_skill":0.0861998388060945,"baseline_adjusted_skill":0.33288220551378433,"baseline_adjusted_score":0.33288220551378433}. Counts: {"cases": 283, "expected": 283, "succeeded": 283, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 283

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/jmmlu-japan-choice / test · target_mass_at_prediction. Population 300 cases / 300 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Choice target mass is a fraction converted to percent; soft targets are not ordinary hard-label accuracy. Diagnostics: {"target_mass_at_prediction":0.8666666666666667,"uniform_accuracy_baseline":0.25,"fixed_answer_accuracy_baseline":0.25,"baseline_adjusted_skill":0.8222222222222223,"baseline_adjusted_score":0.8222222222222223}. Counts: {"cases": 300, "expected": 300, "succeeded": 300, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 300

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/jmteb-retrieval / test · target_mass_at_prediction. Population 300 cases / 300 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Choice target mass is a fraction converted to percent; soft targets are not ordinary hard-label accuracy. Diagnostics: {"target_mass_at_prediction":0.9766666666666667,"uniform_accuracy_baseline":0.20616666666666666,"fixed_answer_accuracy_baseline":0.21333333333333335,"baseline_adjusted_skill":0.9703389830508474,"baseline_adjusted_score":0.9703389830508474}. Counts: {"cases": 300, "expected": 300, "succeeded": 300, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 300

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/jnli-choice / test · target_mass_at_prediction. Population 300 cases / 300 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Choice target mass is a fraction converted to percent; soft targets are not ordinary hard-label accuracy. Diagnostics: {"target_mass_at_prediction":0.9166666666666666,"uniform_accuracy_baseline":0.3333333333333333,"fixed_answer_accuracy_baseline":0.3333333333333333,"baseline_adjusted_skill":0.8749999999999998,"baseline_adjusted_score":0.8749999999999998}. Counts: {"cases": 300, "expected": 300, "succeeded": 300, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 300

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/jsem-choice / test · target_mass_at_prediction. Population 300 cases / 300 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Choice target mass is a fraction converted to percent; soft targets are not ordinary hard-label accuracy. Diagnostics: {"target_mass_at_prediction":0.7866666666666666,"uniform_accuracy_baseline":0.25,"fixed_answer_accuracy_baseline":0.37333333333333335,"baseline_adjusted_skill":0.6595744680851062,"baseline_adjusted_score":0.6595744680851062}. Counts: {"cases": 300, "expected": 300, "succeeded": 300, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 300

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/jsick-sts-score / test · normalized_expected_score_mae. Population 300 cases / 300 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"normalized_expected_score_mae":0.13837466407428947,"normalized_score_rmse":0.17491005419025013,"constant_score_baseline":0.5,"constant_mae_baseline":0.26208333333333333,"baseline_adjusted_skill":0.47202035965294953,"baseline_adjusted_score":0.47202035965294953}. Counts: {"cases": 300, "expected": 300, "succeeded": 300, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 300

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/jsts-score / test · normalized_expected_score_mae. Population 300 cases / 300 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"normalized_expected_score_mae":0.07455207109528508,"normalized_score_rmse":0.0979601678694482,"constant_score_baseline":0.52,"constant_mae_baseline":0.24733333333333335,"baseline_adjusted_skill":0.6985765319597639,"baseline_adjusted_score":0.6985765319597639}. Counts: {"cases": 300, "expected": 300, "succeeded": 300, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 300

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/koniq-score / test · normalized_expected_score_mae. Population 300 cases / 300 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"normalized_expected_score_mae":0.16007694151884322,"normalized_score_rmse":0.18193274083218958,"constant_score_baseline":0.578319964349225,"constant_mae_baseline":0.11212962028602311,"baseline_adjusted_skill":-0.4276062035215571,"baseline_adjusted_score":0}. Counts: {"cases": 300, "expected": 300, "succeeded": 300, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 300

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/kuci-choice / test · target_mass_at_prediction. Population 300 cases / 300 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Choice target mass is a fraction converted to percent; soft targets are not ordinary hard-label accuracy. Diagnostics: {"target_mass_at_prediction":0.8933333333333333,"uniform_accuracy_baseline":0.25,"fixed_answer_accuracy_baseline":0.25,"baseline_adjusted_skill":0.8577777777777778,"baseline_adjusted_score":0.8577777777777778}. Counts: {"cases": 300, "expected": 300, "succeeded": 300, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 300

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/laya__ag_news / test · target_mass_at_prediction. Population 100 cases / 100 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Choice target mass is a fraction converted to percent; soft targets are not ordinary hard-label accuracy. Diagnostics: {"target_mass_at_prediction":0.9,"uniform_accuracy_baseline":0.25,"fixed_answer_accuracy_baseline":0.33,"baseline_adjusted_skill":0.8507462686567167,"baseline_adjusted_score":0.8507462686567167}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/laya__enron_spam / test · binary_brier. Population 100 cases / 100 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"binary_brier":0.04806740967368753,"accuracy":0.93,"positive_recall":0.9,"specificity":0.96,"precision":0.9574468085106383,"balanced_accuracy":0.9299999999999999,"f1":0.9278350515463918,"true_positive":45,"false_positive":2,"false_negative":5,"true_negative":48,"positive_prevalence":0.5,"majority_accuracy_baseline":0.5,"constant_brier_baseline":0.25,"brier_skill":0.8077303613052499,"baseline_adjusted_skill":0.8599999999999999,"baseline_adjusted_score":0.8599999999999999}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/laya__typed_decisions / test · target_mass_at_prediction. Population 100 cases / 148 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Choice target mass is a fraction converted to percent; soft targets are not ordinary hard-label accuracy. Diagnostics: {"target_mass_at_prediction":0.4784246351351351,"uniform_accuracy_baseline":0.23445945945945948,"fixed_answer_accuracy_baseline":0.2623439054054054,"baseline_adjusted_skill":0.2929288205074543,"baseline_adjusted_score":0.2929288205074543}. Counts: {"cases": 100, "expected": 148, "succeeded": 148, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/laya__typed_decisions / test · binary_brier. Population 100 cases / 152 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"binary_brier":0.06354622057604059,"accuracy":0.7166227960526316,"positive_recall":0.8501933025279214,"specificity":0.5894340196288474,"precision":0.6635087789473685,"balanced_accuracy":0.7198136610783844,"f1":0.7453391614282991,"true_positive":63.033334,"false_positive":31.966666,"false_negative":11.106669,"true_negative":45.893330999999996,"positive_prevalence":0.4877631776315789,"majority_accuracy_baseline":0.512236822368421,"constant_brier_baseline":0.09451910039768556,"brier_skill":0.32768910930518536,"baseline_adjusted_skill":0.4396273221567688,"baseline_adjusted_score":0.4396273221567688}. Counts: {"cases": 100, "expected": 152, "succeeded": 152, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/laya__typed_decisions / test · normalized_expected_score_mae. Population 100 cases / 200 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"normalized_expected_score_mae":0.10241473000029773,"normalized_score_rmse":0.13436216665881667,"constant_score_baseline":0.5841668333333333,"constant_mae_baseline":0.21700552708333334,"baseline_adjusted_skill":0.5280547395414084,"baseline_adjusted_score":0.5280547395414084}. Counts: {"cases": 100, "expected": 200, "succeeded": 200, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/lexcomp / test · binary_brier. Population 100 cases / 100 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"binary_brier":0.16612708408676125,"accuracy":0.78,"positive_recall":0.84,"specificity":0.72,"precision":0.75,"balanced_accuracy":0.78,"f1":0.7924528301886793,"true_positive":42,"false_positive":14,"false_negative":8,"true_negative":36,"positive_prevalence":0.5,"majority_accuracy_baseline":0.5,"constant_brier_baseline":0.25,"brier_skill":0.335491663652955,"baseline_adjusted_skill":0.56,"baseline_adjusted_score":0.56}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/logical_entailment / test · binary_brier. Population 100 cases / 100 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"binary_brier":0.05440854192632852,"accuracy":0.92,"positive_recall":0.92,"specificity":0.92,"precision":0.92,"balanced_accuracy":0.92,"f1":0.92,"true_positive":46,"false_positive":4,"false_negative":4,"true_negative":46,"positive_prevalence":0.5,"majority_accuracy_baseline":0.5,"constant_brier_baseline":0.25,"brier_skill":0.7823658322946859,"baseline_adjusted_skill":0.8400000000000001,"baseline_adjusted_score":0.8400000000000001}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/lonli / test · target_mass_at_prediction. Population 100 cases / 100 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Choice target mass is a fraction converted to percent; soft targets are not ordinary hard-label accuracy. Diagnostics: {"target_mass_at_prediction":0.99,"uniform_accuracy_baseline":0.33333333333333326,"fixed_answer_accuracy_baseline":0.39,"baseline_adjusted_skill":0.9836065573770492,"baseline_adjusted_score":0.9836065573770492}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/massive / test · target_mass_at_prediction. Population 100 cases / 200 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Choice target mass is a fraction converted to percent; soft targets are not ordinary hard-label accuracy. Diagnostics: {"target_mass_at_prediction":0.935,"uniform_accuracy_baseline":0.036111111111111115,"fixed_answer_accuracy_baseline":0.08833333333333333,"baseline_adjusted_skill":0.9287020109689215,"baseline_adjusted_score":0.9287020109689215}. Counts: {"cases": 100, "expected": 200, "succeeded": 200, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/mintaka-ja-boolean-noul / test · binary_brier. Population 300 cases / 300 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"binary_brier":0.12088363988645667,"accuracy":0.8333333333333334,"positive_recall":0.8333333333333334,"specificity":0.8333333333333334,"precision":0.8333333333333334,"balanced_accuracy":0.8333333333333334,"f1":0.8333333333333334,"true_positive":125,"false_positive":25,"false_negative":25,"true_negative":125,"positive_prevalence":0.5,"majority_accuracy_baseline":0.5,"constant_brier_baseline":0.25,"brier_skill":0.5164654404541733,"baseline_adjusted_skill":0.6666666666666667,"baseline_adjusted_score":0.6666666666666667}. Counts: {"cases": 300, "expected": 300, "succeeded": 300, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 300

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/miqa / test · target_mass_at_prediction. Population 100 cases / 100 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Choice target mass is a fraction converted to percent; soft targets are not ordinary hard-label accuracy. Diagnostics: {"target_mass_at_prediction":0.97,"uniform_accuracy_baseline":0.5,"fixed_answer_accuracy_baseline":0.56,"baseline_adjusted_skill":0.9318181818181818,"baseline_adjusted_score":0.9318181818181818}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/mmlu-prox-ja-choice / test · target_mass_at_prediction. Population 300 cases / 300 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Choice target mass is a fraction converted to percent; soft targets are not ordinary hard-label accuracy. Diagnostics: {"target_mass_at_prediction":0.6833333333333333,"uniform_accuracy_baseline":0.11021428571428572,"fixed_answer_accuracy_baseline":0.12554761904761905,"baseline_adjusted_skill":0.6378685980341439,"baseline_adjusted_score":0.6378685980341439}. Counts: {"cases": 300, "expected": 300, "succeeded": 300, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 300

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/mmmlu-choice / test · target_mass_at_prediction. Population 300 cases / 300 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Choice target mass is a fraction converted to percent; soft targets are not ordinary hard-label accuracy. Diagnostics: {"target_mass_at_prediction":0.8,"uniform_accuracy_baseline":0.25,"fixed_answer_accuracy_baseline":0.25,"baseline_adjusted_skill":0.7333333333333334,"baseline_adjusted_score":0.7333333333333334}. Counts: {"cases": 300, "expected": 300, "succeeded": 300, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 300

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/mtop / test · target_mass_at_prediction. Population 100 cases / 200 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Choice target mass is a fraction converted to percent; soft targets are not ordinary hard-label accuracy. Diagnostics: {"target_mass_at_prediction":0.92,"uniform_accuracy_baseline":0.049879324215607396,"fixed_answer_accuracy_baseline":0.12442477876106194,"baseline_adjusted_skill":0.9086314938346473,"baseline_adjusted_score":0.9086314938346473}. Counts: {"cases": 100, "expected": 200, "succeeded": 200, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/mvtec-ad-noul / test · binary_brier. Population 300 cases / 300 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"binary_brier":0.1744803510781751,"accuracy":0.7333333333333333,"positive_recall":0.6435185185185185,"specificity":0.9642857142857143,"precision":0.9788732394366197,"balanced_accuracy":0.8039021164021164,"f1":0.776536312849162,"true_positive":139,"false_positive":3,"false_negative":77,"true_negative":81,"positive_prevalence":0.72,"majority_accuracy_baseline":0.72,"constant_brier_baseline":0.2016,"brier_skill":0.13452206806460754,"baseline_adjusted_skill":0.6078042328042328,"baseline_adjusted_score":0.6078042328042328}. Counts: {"cases": 300, "expected": 300, "succeeded": 300, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 300

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/nights-choice / test · target_mass_at_prediction. Population 300 cases / 300 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Choice target mass is a fraction converted to percent; soft targets are not ordinary hard-label accuracy. Diagnostics: {"target_mass_at_prediction":0.9433333333333334,"uniform_accuracy_baseline":0.5,"fixed_answer_accuracy_baseline":0.5033333333333333,"baseline_adjusted_skill":0.8859060402684564,"baseline_adjusted_score":0.8859060402684564}. Counts: {"cases": 300, "expected": 300, "succeeded": 300, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 300

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/nlsat / test · binary_brier. Population 100 cases / 100 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"binary_brier":0.27680869233681993,"accuracy":0.58,"positive_recall":0.16,"specificity":1,"precision":1,"balanced_accuracy":0.58,"f1":0.27586206896551724,"true_positive":8,"false_positive":0,"false_negative":42,"true_negative":50,"positive_prevalence":0.5,"majority_accuracy_baseline":0.5,"constant_brier_baseline":0.25,"brier_skill":-0.10723476934727971,"baseline_adjusted_skill":0.15999999999999992,"baseline_adjusted_score":0.15999999999999992}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/nlvr2-noul / test · binary_brier. Population 300 cases / 300 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"binary_brier":0.0393451856919843,"accuracy":0.9466666666666667,"positive_recall":0.975,"specificity":0.9142857142857143,"precision":0.9285714285714286,"balanced_accuracy":0.9446428571428571,"f1":0.9512195121951219,"true_positive":156,"false_positive":12,"false_negative":4,"true_negative":128,"positive_prevalence":0.5333333333333333,"majority_accuracy_baseline":0.5333333333333333,"constant_brier_baseline":0.2488888888888889,"brier_skill":0.8419166646304203,"baseline_adjusted_skill":0.8892857142857142,"baseline_adjusted_score":0.8892857142857142}. Counts: {"cases": 300, "expected": 300, "succeeded": 300, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 300

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/open_jev__amount-extraction-control-v1 / test · target_mass_at_prediction. Population 36 cases / 36 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Choice target mass is a fraction converted to percent; soft targets are not ordinary hard-label accuracy. Diagnostics: {"target_mass_at_prediction":0.8888888888888888,"uniform_accuracy_baseline":0.23888888888888887,"fixed_answer_accuracy_baseline":0.2916666666666667,"baseline_adjusted_skill":0.8431372549019607,"baseline_adjusted_score":0.8431372549019607}. Counts: {"cases": 36, "expected": 36, "succeeded": 36, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 36

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/open_jev__amount-extraction-control-v1 / test · binary_brier. Population 64 cases / 64 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"binary_brier":0.03632724609725555,"accuracy":0.9375,"positive_recall":0.90625,"specificity":0.96875,"precision":0.9666666666666667,"balanced_accuracy":0.9375,"f1":0.9354838709677419,"true_positive":29,"false_positive":1,"false_negative":3,"true_negative":31,"positive_prevalence":0.5,"majority_accuracy_baseline":0.5,"constant_brier_baseline":0.25,"brier_skill":0.8546910156109778,"baseline_adjusted_skill":0.875,"baseline_adjusted_score":0.875}. Counts: {"cases": 64, "expected": 64, "succeeded": 64, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 64

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/open_jev__browser-control-v1 / test · target_mass_at_prediction. Population 100 cases / 100 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Choice target mass is a fraction converted to percent; soft targets are not ordinary hard-label accuracy. Diagnostics: {"target_mass_at_prediction":1,"uniform_accuracy_baseline":0.33333333333333326,"fixed_answer_accuracy_baseline":0.54,"baseline_adjusted_skill":1,"baseline_adjusted_score":1}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/open_jev__citation-control-v1 / test · target_mass_at_prediction. Population 100 cases / 100 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Choice target mass is a fraction converted to percent; soft targets are not ordinary hard-label accuracy. Diagnostics: {"target_mass_at_prediction":0.85,"uniform_accuracy_baseline":0.33333333333333326,"fixed_answer_accuracy_baseline":0.34,"baseline_adjusted_skill":0.7727272727272728,"baseline_adjusted_score":0.7727272727272728}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/open_jev__context-retention-control-v1 / test · binary_brier. Population 100 cases / 100 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"binary_brier":5.647689911126219e-07,"accuracy":1,"positive_recall":1,"specificity":1,"precision":1,"balanced_accuracy":1,"f1":1,"true_positive":50,"false_positive":0,"false_negative":0,"true_negative":50,"positive_prevalence":0.5,"majority_accuracy_baseline":0.5,"constant_brier_baseline":0.25,"brier_skill":0.9999977409240356,"baseline_adjusted_skill":1,"baseline_adjusted_score":1}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/open_jev__customer-control-v1 / test · target_mass_at_prediction. Population 17 cases / 17 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Choice target mass is a fraction converted to percent; soft targets are not ordinary hard-label accuracy. Diagnostics: {"target_mass_at_prediction":0.8823529411764706,"uniform_accuracy_baseline":0.25,"fixed_answer_accuracy_baseline":0.3235294117647059,"baseline_adjusted_skill":0.8260869565217391,"baseline_adjusted_score":0.8260869565217391}. Counts: {"cases": 17, "expected": 17, "succeeded": 17, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 17

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/open_jev__customer-control-v1 / test · binary_brier. Population 33 cases / 33 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"binary_brier":0.0009955801169891998,"accuracy":1,"positive_recall":1,"specificity":1,"precision":1,"balanced_accuracy":1,"f1":1,"true_positive":16,"false_positive":0,"false_negative":0,"true_negative":17,"positive_prevalence":0.48484848484848486,"majority_accuracy_baseline":0.5151515151515151,"constant_brier_baseline":0.2497704315886134,"brier_skill":0.9960140193110248,"baseline_adjusted_skill":1,"baseline_adjusted_score":1}. Counts: {"cases": 33, "expected": 33, "succeeded": 33, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 33

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/open_jev__customer-control-v1 / test · normalized_expected_score_mae. Population 50 cases / 50 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"normalized_expected_score_mae":0.1235129741304708,"normalized_score_rmse":0.2756724245424017,"constant_score_baseline":0.25,"constant_mae_baseline":0.33,"baseline_adjusted_skill":0.6257182602106945,"baseline_adjusted_score":0.6257182602106945}. Counts: {"cases": 50, "expected": 50, "succeeded": 50, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 50

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/open_jev__drone-control-v1 / test · target_mass_at_prediction. Population 32 cases / 32 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Choice target mass is a fraction converted to percent; soft targets are not ordinary hard-label accuracy. Diagnostics: {"target_mass_at_prediction":1,"uniform_accuracy_baseline":0.178125,"fixed_answer_accuracy_baseline":0.2875,"baseline_adjusted_skill":1,"baseline_adjusted_score":1}. Counts: {"cases": 32, "expected": 32, "succeeded": 32, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 32

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/open_jev__drone-control-v1 / test · binary_brier. Population 34 cases / 34 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"binary_brier":1.1975901063693981e-06,"accuracy":1,"positive_recall":1,"specificity":1,"precision":1,"balanced_accuracy":1,"f1":1,"true_positive":17,"false_positive":0,"false_negative":0,"true_negative":17,"positive_prevalence":0.5,"majority_accuracy_baseline":0.5,"constant_brier_baseline":0.25,"brier_skill":0.9999952096395746,"baseline_adjusted_skill":1,"baseline_adjusted_score":1}. Counts: {"cases": 34, "expected": 34, "succeeded": 34, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 34

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/open_jev__drone-control-v1 / test · normalized_expected_score_mae. Population 34 cases / 34 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"normalized_expected_score_mae":0.05807996199297264,"normalized_score_rmse":0.08306384218138756,"constant_score_baseline":0.5,"constant_mae_baseline":0.14705882352941177,"baseline_adjusted_skill":0.6050562584477861,"baseline_adjusted_score":0.6050562584477861}. Counts: {"cases": 34, "expected": 34, "succeeded": 34, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 34

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/open_jev__email-selection-control-v1 / test · target_mass_at_prediction. Population 48 cases / 48 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Choice target mass is a fraction converted to percent; soft targets are not ordinary hard-label accuracy. Diagnostics: {"target_mass_at_prediction":1,"uniform_accuracy_baseline":0.13194444444444445,"fixed_answer_accuracy_baseline":0.6006944444444444,"baseline_adjusted_skill":1,"baseline_adjusted_score":1}. Counts: {"cases": 48, "expected": 48, "succeeded": 48, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 48

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/open_jev__email-selection-control-v1 / test · binary_brier. Population 52 cases / 52 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"binary_brier":2.0889265122043423e-08,"accuracy":1,"positive_recall":1,"specificity":1,"precision":1,"balanced_accuracy":1,"f1":1,"true_positive":26,"false_positive":0,"false_negative":0,"true_negative":26,"positive_prevalence":0.5,"majority_accuracy_baseline":0.5,"constant_brier_baseline":0.25,"brier_skill":0.9999999164429395,"baseline_adjusted_skill":1,"baseline_adjusted_score":1}. Counts: {"cases": 52, "expected": 52, "succeeded": 52, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 52

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/open_jev__entity-alignment-control-v1 / test · binary_brier. Population 75 cases / 75 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"binary_brier":5.990048097962631e-07,"accuracy":1,"positive_recall":1,"specificity":1,"precision":1,"balanced_accuracy":1,"f1":1,"true_positive":39,"false_positive":0,"false_negative":0,"true_negative":36,"positive_prevalence":0.52,"majority_accuracy_baseline":0.52,"constant_brier_baseline":0.24960000000000004,"brier_skill":0.9999976001409864,"baseline_adjusted_skill":1,"baseline_adjusted_score":1}. Counts: {"cases": 75, "expected": 75, "succeeded": 75, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 75

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/open_jev__entity-alignment-control-v1 / test · normalized_expected_score_mae. Population 25 cases / 25 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"normalized_expected_score_mae":0.031744397011244645,"normalized_score_rmse":0.07200152903200976,"constant_score_baseline":0.5,"constant_mae_baseline":0.28,"baseline_adjusted_skill":0.8866271535312691,"baseline_adjusted_score":0.8866271535312691}. Counts: {"cases": 25, "expected": 25, "succeeded": 25, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 25

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/open_jev__ir-control-v1 / test · target_mass_at_prediction. Population 17 cases / 17 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Choice target mass is a fraction converted to percent; soft targets are not ordinary hard-label accuracy. Diagnostics: {"target_mass_at_prediction":0.9117647058823529,"uniform_accuracy_baseline":0.25,"fixed_answer_accuracy_baseline":0.2647058823529412,"baseline_adjusted_skill":0.8799999999999998,"baseline_adjusted_score":0.8799999999999998}. Counts: {"cases": 17, "expected": 17, "succeeded": 17, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 17

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/open_jev__ir-control-v1 / test · binary_brier. Population 28 cases / 28 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"binary_brier":0.029996614032655546,"accuracy":0.9642857142857143,"positive_recall":1,"specificity":0.9285714285714286,"precision":0.9333333333333333,"balanced_accuracy":0.9642857142857143,"f1":0.9655172413793104,"true_positive":14,"false_positive":1,"false_negative":0,"true_negative":13,"positive_prevalence":0.5,"majority_accuracy_baseline":0.5,"constant_brier_baseline":0.25,"brier_skill":0.8800135438693778,"baseline_adjusted_skill":0.9285714285714286,"baseline_adjusted_score":0.9285714285714286}. Counts: {"cases": 28, "expected": 28, "succeeded": 28, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 28

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/open_jev__ir-control-v1 / test · normalized_expected_score_mae. Population 55 cases / 55 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"normalized_expected_score_mae":0.017406223187637952,"normalized_score_rmse":0.037536259349378494,"constant_score_baseline":0.3333333333333333,"constant_mae_baseline":0.1696969696969697,"baseline_adjusted_skill":0.897427613358562,"baseline_adjusted_score":0.897427613358562}. Counts: {"cases": 55, "expected": 55, "succeeded": 55, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 55

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/open_jev__mailroom-control-v1 / test · target_mass_at_prediction. Population 15 cases / 15 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Choice target mass is a fraction converted to percent; soft targets are not ordinary hard-label accuracy. Diagnostics: {"target_mass_at_prediction":1,"uniform_accuracy_baseline":0.15555555555555553,"fixed_answer_accuracy_baseline":0.2,"baseline_adjusted_skill":1,"baseline_adjusted_score":1}. Counts: {"cases": 15, "expected": 15, "succeeded": 15, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 15

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/open_jev__mailroom-control-v1 / test · binary_brier. Population 85 cases / 85 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"binary_brier":0.014288767601365044,"accuracy":0.9764705882352941,"positive_recall":1,"specificity":0.95,"precision":0.9574468085106383,"balanced_accuracy":0.975,"f1":0.9782608695652174,"true_positive":45,"false_positive":2,"false_negative":0,"true_negative":38,"positive_prevalence":0.5294117647058824,"majority_accuracy_baseline":0.5294117647058824,"constant_brier_baseline":0.2491349480968858,"brier_skill":0.9426464744889653,"baseline_adjusted_skill":0.95,"baseline_adjusted_score":0.95}. Counts: {"cases": 85, "expected": 85, "succeeded": 85, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 85

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/open_jev__painting-geometry-v1 / test · target_mass_at_prediction. Population 25 cases / 25 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Choice target mass is a fraction converted to percent; soft targets are not ordinary hard-label accuracy. Diagnostics: {"target_mass_at_prediction":1,"uniform_accuracy_baseline":0.07805555555555556,"fixed_answer_accuracy_baseline":0.24,"baseline_adjusted_skill":1,"baseline_adjusted_score":1}. Counts: {"cases": 25, "expected": 25, "succeeded": 25, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 25

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/open_jev__painting-geometry-v1 / test · binary_brier. Population 50 cases / 50 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"binary_brier":2.6196586886032655e-06,"accuracy":1,"positive_recall":1,"specificity":1,"precision":1,"balanced_accuracy":1,"f1":1,"true_positive":35,"false_positive":0,"false_negative":0,"true_negative":15,"positive_prevalence":0.7,"majority_accuracy_baseline":0.7,"constant_brier_baseline":0.21,"brier_skill":0.9999875254348162,"baseline_adjusted_skill":1,"baseline_adjusted_score":1}. Counts: {"cases": 50, "expected": 50, "succeeded": 50, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 50

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/open_jev__painting-geometry-v1 / test · normalized_expected_score_mae. Population 25 cases / 25 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"normalized_expected_score_mae":0.0029277405869167373,"normalized_score_rmse":0.004030323143889149,"constant_score_baseline":0.5,"constant_mae_baseline":0.28,"baseline_adjusted_skill":0.9895437836181545,"baseline_adjusted_score":0.9895437836181545}. Counts: {"cases": 25, "expected": 25, "succeeded": 25, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 25

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/open_jev__phone-extraction-control-v1 / test · target_mass_at_prediction. Population 79 cases / 79 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Choice target mass is a fraction converted to percent; soft targets are not ordinary hard-label accuracy. Diagnostics: {"target_mass_at_prediction":0.8607594936708861,"uniform_accuracy_baseline":0.24240506329113923,"fixed_answer_accuracy_baseline":0.4177215189873418,"baseline_adjusted_skill":0.7608695652173914,"baseline_adjusted_score":0.7608695652173914}. Counts: {"cases": 79, "expected": 79, "succeeded": 79, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 79

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/open_jev__phone-extraction-control-v1 / test · binary_brier. Population 21 cases / 21 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"binary_brier":4.4993328321790745e-08,"accuracy":1,"positive_recall":1,"specificity":1,"precision":1,"balanced_accuracy":1,"f1":1,"true_positive":11,"false_positive":0,"false_negative":0,"true_negative":10,"positive_prevalence":0.5238095238095238,"majority_accuracy_baseline":0.5238095238095238,"constant_brier_baseline":0.2494331065759637,"brier_skill":0.9999998196176565,"baseline_adjusted_skill":1,"baseline_adjusted_score":1}. Counts: {"cases": 21, "expected": 21, "succeeded": 21, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 21

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/open_jev__reasoning-control-v1 / test · target_mass_at_prediction. Population 33 cases / 33 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Choice target mass is a fraction converted to percent; soft targets are not ordinary hard-label accuracy. Diagnostics: {"target_mass_at_prediction":0.9393939393939394,"uniform_accuracy_baseline":0.32373737373737377,"fixed_answer_accuracy_baseline":0.3939393939393939,"baseline_adjusted_skill":0.9000000000000001,"baseline_adjusted_score":0.9000000000000001}. Counts: {"cases": 33, "expected": 33, "succeeded": 33, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 33

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/open_jev__reasoning-control-v1 / test · binary_brier. Population 33 cases / 33 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"binary_brier":0.09236842328560274,"accuracy":0.8787878787878788,"positive_recall":0.8235294117647058,"specificity":0.9375,"precision":0.9333333333333333,"balanced_accuracy":0.8805147058823529,"f1":0.875,"true_positive":14,"false_positive":1,"false_negative":3,"true_negative":15,"positive_prevalence":0.5151515151515151,"majority_accuracy_baseline":0.5151515151515151,"constant_brier_baseline":0.2497704315886134,"brier_skill":0.6301867170660979,"baseline_adjusted_skill":0.7610294117647058,"baseline_adjusted_score":0.7610294117647058}. Counts: {"cases": 33, "expected": 33, "succeeded": 33, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 33

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/open_jev__reasoning-control-v1 / test · normalized_expected_score_mae. Population 34 cases / 34 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"normalized_expected_score_mae":0.013354410554721988,"normalized_score_rmse":0.03865262888505102,"constant_score_baseline":0.7083333333333333,"constant_mae_baseline":0.3431372549019608,"baseline_adjusted_skill":0.9610814320976674,"baseline_adjusted_score":0.9610814320976674}. Counts: {"cases": 34, "expected": 34, "succeeded": 34, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 34

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/open_jev__silent-failure-control-v1 / test · binary_brier. Population 100 cases / 100 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"binary_brier":0.0010917549244628022,"accuracy":1,"positive_recall":1,"specificity":1,"precision":1,"balanced_accuracy":1,"f1":1,"true_positive":50,"false_positive":0,"false_negative":0,"true_negative":50,"positive_prevalence":0.5,"majority_accuracy_baseline":0.5,"constant_brier_baseline":0.25,"brier_skill":0.9956329803021488,"baseline_adjusted_skill":1,"baseline_adjusted_score":1}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/open_jev__snake-v1 / test · target_mass_at_prediction. Population 25 cases / 25 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Choice target mass is a fraction converted to percent; soft targets are not ordinary hard-label accuracy. Diagnostics: {"target_mass_at_prediction":0.8133333333333332,"uniform_accuracy_baseline":0.33333333333333326,"fixed_answer_accuracy_baseline":0.35333333333333333,"baseline_adjusted_skill":0.7113402061855668,"baseline_adjusted_score":0.7113402061855668}. Counts: {"cases": 25, "expected": 25, "succeeded": 25, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 25

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/open_jev__snake-v1 / test · binary_brier. Population 75 cases / 75 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"binary_brier":0.056880444046686914,"accuracy":0.92,"positive_recall":0.8648648648648649,"specificity":0.9736842105263158,"precision":0.9696969696969697,"balanced_accuracy":0.9192745376955904,"f1":0.9142857142857143,"true_positive":32,"false_positive":1,"false_negative":5,"true_negative":37,"positive_prevalence":0.49333333333333335,"majority_accuracy_baseline":0.5066666666666666,"constant_brier_baseline":0.24995555555555554,"brier_skill":0.7724377683053956,"baseline_adjusted_skill":0.8385490753911808,"baseline_adjusted_score":0.8385490753911808}. Counts: {"cases": 75, "expected": 75, "succeeded": 75, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 75

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/open_jev__sponsor-segment-control-v1 / test · target_mass_at_prediction. Population 100 cases / 100 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Choice target mass is a fraction converted to percent; soft targets are not ordinary hard-label accuracy. Diagnostics: {"target_mass_at_prediction":1,"uniform_accuracy_baseline":0.14285714285714285,"fixed_answer_accuracy_baseline":0.22,"baseline_adjusted_skill":1,"baseline_adjusted_score":1}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/open_jev__tic_tac_toe-v1 / test · target_mass_at_prediction. Population 100 cases / 100 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Choice target mass is a fraction converted to percent; soft targets are not ordinary hard-label accuracy. Diagnostics: {"target_mass_at_prediction":0.39,"uniform_accuracy_baseline":0.32416666666666666,"fixed_answer_accuracy_baseline":0.34,"baseline_adjusted_skill":0.07575757575757575,"baseline_adjusted_score":0.07575757575757575}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/open_jev__vizdoom-basic-v1 / test · binary_brier. Population 33 cases / 33 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"binary_brier":1.1838271635626526e-06,"accuracy":1,"positive_recall":1,"specificity":1,"precision":1,"balanced_accuracy":1,"f1":1,"true_positive":16,"false_positive":0,"false_negative":0,"true_negative":17,"positive_prevalence":0.48484848484848486,"majority_accuracy_baseline":0.5151515151515151,"constant_brier_baseline":0.2497704315886134,"brier_skill":0.99999526033904,"baseline_adjusted_skill":1,"baseline_adjusted_score":1}. Counts: {"cases": 33, "expected": 33, "succeeded": 33, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 33

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/open_jev__vizdoom-basic-v1 / test · normalized_expected_score_mae. Population 33 cases / 33 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"normalized_expected_score_mae":0.0017792138242399985,"normalized_score_rmse":0.004779509250860236,"constant_score_baseline":0.5,"constant_mae_baseline":0.2196969696969697,"baseline_adjusted_skill":0.9919015094896663,"baseline_adjusted_score":0.9919015094896663}. Counts: {"cases": 33, "expected": 33, "succeeded": 33, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 33

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/open_jev__workflow-controls-v1__agent_trace_observability / test · binary_brier. Population 100 cases / 100 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"binary_brier":0.09986091823051818,"accuracy":0.88,"positive_recall":1,"specificity":0.8461538461538461,"precision":0.6470588235294118,"balanced_accuracy":0.9230769230769231,"f1":0.7857142857142857,"true_positive":22,"false_positive":12,"false_negative":0,"true_negative":66,"positive_prevalence":0.22,"majority_accuracy_baseline":0.78,"constant_brier_baseline":0.1716,"brier_skill":0.4180599170715724,"baseline_adjusted_skill":0.8461538461538463,"baseline_adjusted_score":0.8461538461538463}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/open_jev__workflow-controls-v1__customer_service / test · binary_brier. Population 100 cases / 100 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"binary_brier":0.08070555173996372,"accuracy":0.87,"positive_recall":0.6486486486486487,"specificity":1,"precision":1,"balanced_accuracy":0.8243243243243243,"f1":0.7868852459016393,"true_positive":24,"false_positive":0,"false_negative":13,"true_negative":63,"positive_prevalence":0.37,"majority_accuracy_baseline":0.63,"constant_brier_baseline":0.23310000000000003,"brier_skill":0.6537728368083925,"baseline_adjusted_skill":0.6486486486486487,"baseline_adjusted_score":0.6486486486486487}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/open_jev__workflow-controls-v1__invoice_processing / test · binary_brier. Population 100 cases / 100 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"binary_brier":0.04747151172693969,"accuracy":0.94,"positive_recall":0.8461538461538461,"specificity":1,"precision":1,"balanced_accuracy":0.9230769230769231,"f1":0.9166666666666666,"true_positive":33,"false_positive":0,"false_negative":6,"true_negative":61,"positive_prevalence":0.39,"majority_accuracy_baseline":0.61,"constant_brier_baseline":0.2379,"brier_skill":0.8004560246870968,"baseline_adjusted_skill":0.8461538461538463,"baseline_adjusted_score":0.8461538461538463}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/open_jev__workflow-controls-v1__security_incidents / test · binary_brier. Population 100 cases / 100 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"binary_brier":0.05344993881589085,"accuracy":0.92,"positive_recall":0.9032258064516129,"specificity":0.927536231884058,"precision":0.8484848484848485,"balanced_accuracy":0.9153810191678354,"f1":0.875,"true_positive":28,"false_positive":5,"false_negative":3,"true_negative":64,"positive_prevalence":0.31,"majority_accuracy_baseline":0.69,"constant_brier_baseline":0.21389999999999998,"brier_skill":0.7501171630860642,"baseline_adjusted_skill":0.8307620383356709,"baseline_adjusted_score":0.8307620383356709}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/openbookqa / test · target_mass_at_prediction. Population 100 cases / 100 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Choice target mass is a fraction converted to percent; soft targets are not ordinary hard-label accuracy. Diagnostics: {"target_mass_at_prediction":0.99,"uniform_accuracy_baseline":0.25,"fixed_answer_accuracy_baseline":0.28,"baseline_adjusted_skill":0.986111111111111,"baseline_adjusted_score":0.986111111111111}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/patent_similarity / test · normalized_expected_score_mae. Population 100 cases / 100 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"normalized_expected_score_mae":0.11202758720903777,"normalized_score_rmse":0.14296439281221443,"constant_score_baseline":0.25,"constant_mae_baseline":0.2325,"baseline_adjusted_skill":0.5181609152299451,"baseline_adjusted_score":0.5181609152299451}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/paws / test · binary_brier. Population 100 cases / 100 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"binary_brier":0.07216010783289431,"accuracy":0.9,"positive_recall":0.88,"specificity":0.92,"precision":0.9166666666666666,"balanced_accuracy":0.9,"f1":0.8979591836734694,"true_positive":44,"false_positive":4,"false_negative":6,"true_negative":46,"positive_prevalence":0.5,"majority_accuracy_baseline":0.5,"constant_brier_baseline":0.25,"brier_skill":0.7113595686684228,"baseline_adjusted_skill":0.8,"baseline_adjusted_score":0.8}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/plane / test · binary_brier. Population 100 cases / 100 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"binary_brier":0.3145433676843672,"accuracy":0.61,"positive_recall":0.84,"specificity":0.38,"precision":0.5753424657534246,"balanced_accuracy":0.61,"f1":0.6829268292682927,"true_positive":42,"false_positive":31,"false_negative":8,"true_negative":19,"positive_prevalence":0.5,"majority_accuracy_baseline":0.5,"constant_brier_baseline":0.25,"brier_skill":-0.25817347073746877,"baseline_adjusted_skill":0.21999999999999997,"baseline_adjusted_score":0.21999999999999997}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/poem_sentiment / test · target_mass_at_prediction. Population 100 cases / 100 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Choice target mass is a fraction converted to percent; soft targets are not ordinary hard-label accuracy. Diagnostics: {"target_mass_at_prediction":0.87,"uniform_accuracy_baseline":0.25,"fixed_answer_accuracy_baseline":0.65,"baseline_adjusted_skill":0.6285714285714286,"baseline_adjusted_score":0.6285714285714286}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/pope-noul / test · binary_brier. Population 300 cases / 300 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"binary_brier":0.08411436585055698,"accuracy":0.9,"positive_recall":0.8266666666666667,"specificity":0.9733333333333334,"precision":0.96875,"balanced_accuracy":0.9,"f1":0.8920863309352518,"true_positive":124,"false_positive":4,"false_negative":26,"true_negative":146,"positive_prevalence":0.5,"majority_accuracy_baseline":0.5,"constant_brier_baseline":0.25,"brier_skill":0.6635425365977721,"baseline_adjusted_skill":0.8,"baseline_adjusted_score":0.8}. Counts: {"cases": 300, "expected": 300, "succeeded": 300, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 300

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/qasper / test · binary_brier. Population 100 cases / 100 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"binary_brier":0.0665995216503909,"accuracy":0.91,"positive_recall":0.94,"specificity":0.88,"precision":0.8867924528301887,"balanced_accuracy":0.9099999999999999,"f1":0.912621359223301,"true_positive":47,"false_positive":6,"false_negative":3,"true_negative":44,"positive_prevalence":0.5,"majority_accuracy_baseline":0.5,"constant_brier_baseline":0.25,"brier_skill":0.7336019133984364,"baseline_adjusted_skill":0.8199999999999998,"baseline_adjusted_score":0.8199999999999998}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/quartz / test · target_mass_at_prediction. Population 100 cases / 100 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Choice target mass is a fraction converted to percent; soft targets are not ordinary hard-label accuracy. Diagnostics: {"target_mass_at_prediction":0.96,"uniform_accuracy_baseline":0.5,"fixed_answer_accuracy_baseline":0.58,"baseline_adjusted_skill":0.9047619047619047,"baseline_adjusted_score":0.9047619047619047}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/robust_lr / test · target_mass_at_prediction. Population 100 cases / 100 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Choice target mass is a fraction converted to percent; soft targets are not ordinary hard-label accuracy. Diagnostics: {"target_mass_at_prediction":0.84,"uniform_accuracy_baseline":0.33333333333333326,"fixed_answer_accuracy_baseline":0.4,"baseline_adjusted_skill":0.7333333333333333,"baseline_adjusted_score":0.7333333333333333}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/ruletaker / test · binary_brier. Population 100 cases / 100 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"binary_brier":0.0006471918478728844,"accuracy":1,"positive_recall":1,"specificity":1,"precision":1,"balanced_accuracy":1,"f1":1,"true_positive":50,"false_positive":0,"false_negative":0,"true_negative":50,"positive_prevalence":0.5,"majority_accuracy_baseline":0.5,"constant_brier_baseline":0.25,"brier_skill":0.9974112326085085,"baseline_adjusted_skill":1,"baseline_adjusted_score":1}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/rvlcdip-choice / test · target_mass_at_prediction. Population 300 cases / 300 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Choice target mass is a fraction converted to percent; soft targets are not ordinary hard-label accuracy. Diagnostics: {"target_mass_at_prediction":0.7133333333333334,"uniform_accuracy_baseline":0.0625,"fixed_answer_accuracy_baseline":0.10666666666666667,"baseline_adjusted_skill":0.6791044776119404,"baseline_adjusted_score":0.6791044776119404}. Counts: {"cases": 300, "expected": 300, "succeeded": 300, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 300

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/s1mb-generalization-contextual-choice / test · target_mass_at_prediction. Population 100 cases / 100 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Choice target mass is a fraction converted to percent; soft targets are not ordinary hard-label accuracy. Diagnostics: {"target_mass_at_prediction":1,"uniform_accuracy_baseline":0.20851190476190473,"fixed_answer_accuracy_baseline":0.22333333333333333,"baseline_adjusted_skill":1,"baseline_adjusted_score":1}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/s1mb-generalization-contextual-noul / test · binary_brier. Population 100 cases / 100 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"binary_brier":0.0022061450272253132,"accuracy":1,"positive_recall":1,"specificity":1,"precision":1,"balanced_accuracy":1,"f1":1,"true_positive":50,"false_positive":0,"false_negative":0,"true_negative":50,"positive_prevalence":0.5,"majority_accuracy_baseline":0.5,"constant_brier_baseline":0.25,"brier_skill":0.9911754198910987,"baseline_adjusted_skill":1,"baseline_adjusted_score":1}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/s1mb-generalization-contextual-score / test · normalized_expected_score_mae. Population 100 cases / 100 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"normalized_expected_score_mae":0.015236639612342893,"normalized_score_rmse":0.04723782640793688,"constant_score_baseline":0.5,"constant_mae_baseline":0.3,"baseline_adjusted_skill":0.9492112012921904,"baseline_adjusted_score":0.9492112012921904}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/s1mb-generalization-diverse-choice / test · target_mass_at_prediction. Population 100 cases / 100 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Choice target mass is a fraction converted to percent; soft targets are not ordinary hard-label accuracy. Diagnostics: {"target_mass_at_prediction":0.99,"uniform_accuracy_baseline":0.22851190476190475,"fixed_answer_accuracy_baseline":0.24333333333333332,"baseline_adjusted_skill":0.986784140969163,"baseline_adjusted_score":0.986784140969163}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/s1mb-generalization-diverse-noul / test · binary_brier. Population 100 cases / 100 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"binary_brier":0.00026474570743171286,"accuracy":1,"positive_recall":1,"specificity":1,"precision":1,"balanced_accuracy":1,"f1":1,"true_positive":50,"false_positive":0,"false_negative":0,"true_negative":50,"positive_prevalence":0.5,"majority_accuracy_baseline":0.5,"constant_brier_baseline":0.25,"brier_skill":0.9989410171702732,"baseline_adjusted_skill":1,"baseline_adjusted_score":1}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/s1mb-generalization-diverse-score / test · normalized_expected_score_mae. Population 100 cases / 100 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"normalized_expected_score_mae":0.013378183832411263,"normalized_score_rmse":0.029563599806815784,"constant_score_baseline":0.4222222222222222,"constant_mae_baseline":0.3042222222222222,"baseline_adjusted_skill":0.956024961836486,"baseline_adjusted_score":0.956024961836486}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/scicite / test · target_mass_at_prediction. Population 100 cases / 100 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Choice target mass is a fraction converted to percent; soft targets are not ordinary hard-label accuracy. Diagnostics: {"target_mass_at_prediction":0.77,"uniform_accuracy_baseline":0.33333333333333326,"fixed_answer_accuracy_baseline":0.38,"baseline_adjusted_skill":0.6290322580645161,"baseline_adjusted_score":0.6290322580645161}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/scienceqa-choice / test · target_mass_at_prediction. Population 300 cases / 300 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Choice target mass is a fraction converted to percent; soft targets are not ordinary hard-label accuracy. Diagnostics: {"target_mass_at_prediction":0.9866666666666667,"uniform_accuracy_baseline":0.3467777777777778,"fixed_answer_accuracy_baseline":0.36,"baseline_adjusted_skill":0.9791666666666667,"baseline_adjusted_score":0.9791666666666667}. Counts: {"cases": 300, "expected": 300, "succeeded": 300, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 300

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/scitail / test · binary_brier. Population 100 cases / 100 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"binary_brier":0.019907528583464375,"accuracy":0.98,"positive_recall":1,"specificity":0.96,"precision":0.9615384615384616,"balanced_accuracy":0.98,"f1":0.9803921568627451,"true_positive":50,"false_positive":2,"false_negative":0,"true_negative":48,"positive_prevalence":0.5,"majority_accuracy_baseline":0.5,"constant_brier_baseline":0.25,"brier_skill":0.9203698856661425,"baseline_adjusted_skill":0.96,"baseline_adjusted_score":0.96}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/scone / test · target_mass_at_prediction. Population 100 cases / 100 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Choice target mass is a fraction converted to percent; soft targets are not ordinary hard-label accuracy. Diagnostics: {"target_mass_at_prediction":0.96,"uniform_accuracy_baseline":0.33333333333333326,"fixed_answer_accuracy_baseline":0.5,"baseline_adjusted_skill":0.9199999999999999,"baseline_adjusted_score":0.9199999999999999}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/sdoh_nli / test · binary_brier. Population 100 cases / 100 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"binary_brier":0.02506508733949101,"accuracy":0.96,"positive_recall":0.92,"specificity":1,"precision":1,"balanced_accuracy":0.96,"f1":0.9583333333333334,"true_positive":46,"false_positive":0,"false_negative":4,"true_negative":50,"positive_prevalence":0.5,"majority_accuracy_baseline":0.5,"constant_brier_baseline":0.25,"brier_skill":0.899739650642036,"baseline_adjusted_skill":0.9199999999999999,"baseline_adjusted_score":0.9199999999999999}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/sgd / test · target_mass_at_prediction. Population 100 cases / 100 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Choice target mass is a fraction converted to percent; soft targets are not ordinary hard-label accuracy. Diagnostics: {"target_mass_at_prediction":0.61,"uniform_accuracy_baseline":0.3533333333333333,"fixed_answer_accuracy_baseline":0.41,"baseline_adjusted_skill":0.3389830508474576,"baseline_adjusted_score":0.3389830508474576}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/snli / test · target_mass_at_prediction. Population 100 cases / 100 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Choice target mass is a fraction converted to percent; soft targets are not ordinary hard-label accuracy. Diagnostics: {"target_mass_at_prediction":0.98,"uniform_accuracy_baseline":0.33333333333333326,"fixed_answer_accuracy_baseline":0.38,"baseline_adjusted_skill":0.9677419354838709,"baseline_adjusted_score":0.9677419354838709}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/sobaco-culture-choice / test · target_mass_at_prediction. Population 300 cases / 300 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Choice target mass is a fraction converted to percent; soft targets are not ordinary hard-label accuracy. Diagnostics: {"target_mass_at_prediction":0.7833333333333333,"uniform_accuracy_baseline":0.3333333333333333,"fixed_answer_accuracy_baseline":0.3333333333333333,"baseline_adjusted_skill":0.6749999999999999,"baseline_adjusted_score":0.6749999999999999}. Counts: {"cases": 300, "expected": 300, "succeeded": 300, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 300

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/spartqa / test · target_mass_at_prediction. Population 100 cases / 100 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Choice target mass is a fraction converted to percent; soft targets are not ordinary hard-label accuracy. Diagnostics: {"target_mass_at_prediction":0.96,"uniform_accuracy_baseline":0.25,"fixed_answer_accuracy_baseline":0.38,"baseline_adjusted_skill":0.9354838709677419,"baseline_adjusted_score":0.9354838709677419}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/stepgame / test · target_mass_at_prediction. Population 100 cases / 100 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Choice target mass is a fraction converted to percent; soft targets are not ordinary hard-label accuracy. Diagnostics: {"target_mass_at_prediction":0.78,"uniform_accuracy_baseline":0.1111111111111111,"fixed_answer_accuracy_baseline":0.16,"baseline_adjusted_skill":0.7380952380952381,"baseline_adjusted_score":0.7380952380952381}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/sugarcrepe-choice / test · target_mass_at_prediction. Population 300 cases / 300 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Choice target mass is a fraction converted to percent; soft targets are not ordinary hard-label accuracy. Diagnostics: {"target_mass_at_prediction":0.98,"uniform_accuracy_baseline":0.5,"fixed_answer_accuracy_baseline":0.58,"baseline_adjusted_skill":0.9523809523809523,"baseline_adjusted_score":0.9523809523809523}. Counts: {"cases": 300, "expected": 300, "succeeded": 300, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 300

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/synthetic_relevance__nanobeir__NanoArguAna / test · binary_brier. Population 400 cases / 400 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"binary_brier":0.033305543566917256,"accuracy":0.8528749999999999,"positive_recall":0.6064601186552406,"specificity":0.9105352460280735,"precision":0.6133333333333333,"balanced_accuracy":0.758497682341657,"f1":0.6098773616175008,"true_positive":46,"false_positive":29,"false_negative":29.85,"true_negative":295.15,"positive_prevalence":0.189625,"majority_accuracy_baseline":0.8103750000000001,"constant_brier_baseline":0.07612360937500001,"brier_skill":0.562480762008439,"baseline_adjusted_skill":0.5169953646833141,"baseline_adjusted_score":0.5169953646833141}. Counts: {"cases": 400, "expected": 400, "succeeded": 400, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 400

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/synthetic_relevance__nanobeir__NanoArguAna / test · normalized_expected_score_mae. Population 400 cases / 400 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"normalized_expected_score_mae":0.14467997161829596,"normalized_score_rmse":0.1745764819558717,"constant_score_baseline":0,"constant_mae_baseline":0.21375,"baseline_adjusted_skill":0.32313463570387857,"baseline_adjusted_score":0.32313463570387857}. Counts: {"cases": 400, "expected": 400, "succeeded": 400, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 400

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/synthetic_relevance__nanobeir__NanoDBPedia / test · binary_brier. Population 400 cases / 400 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"binary_brier":0.05952939379973932,"accuracy":0.8028749999999999,"positive_recall":0.406938584153774,"specificity":0.9468212033407192,"precision":0.735593220338983,"balanced_accuracy":0.6768798937472466,"f1":0.5239963779052219,"true_positive":43.4,"false_positive":15.6,"false_negative":63.25,"true_negative":277.75,"positive_prevalence":0.266625,"majority_accuracy_baseline":0.733375,"constant_brier_baseline":0.115654859375,"brier_skill":0.4852841106596234,"baseline_adjusted_skill":0.3537597874944931,"baseline_adjusted_score":0.3537597874944931}. Counts: {"cases": 400, "expected": 400, "succeeded": 400, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 400

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/synthetic_relevance__nanobeir__NanoDBPedia / test · normalized_expected_score_mae. Population 400 cases / 400 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"normalized_expected_score_mae":0.1588308972990764,"normalized_score_rmse":0.22412485671110782,"constant_score_baseline":0.25,"constant_mae_baseline":0.26875,"baseline_adjusted_skill":0.4090013123755296,"baseline_adjusted_score":0.4090013123755296}. Counts: {"cases": 400, "expected": 400, "succeeded": 400, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 400

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/synthetic_relevance__nanobeir__NanoFiQA2018 / test · binary_brier. Population 400 cases / 400 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"binary_brier":0.039957650493237115,"accuracy":0.823125,"positive_recall":0.7114093959731543,"specificity":0.8664353859496965,"precision":0.673728813559322,"balanced_accuracy":0.7889223909614254,"f1":0.6920565832426551,"true_positive":79.5,"false_positive":38.5,"false_negative":32.25,"true_negative":249.75,"positive_prevalence":0.279375,"majority_accuracy_baseline":0.7206250000000001,"constant_brier_baseline":0.11315585937500001,"brier_skill":0.6468795278128998,"baseline_adjusted_skill":0.5778447819228507,"baseline_adjusted_score":0.5778447819228507}. Counts: {"cases": 400, "expected": 400, "succeeded": 400, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 400

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/synthetic_relevance__nanobeir__NanoFiQA2018 / test · normalized_expected_score_mae. Population 400 cases / 400 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"normalized_expected_score_mae":0.13649079114546875,"normalized_score_rmse":0.17381178872146508,"constant_score_baseline":0.25,"constant_mae_baseline":0.264375,"baseline_adjusted_skill":0.4837227758091017,"baseline_adjusted_score":0.4837227758091017}. Counts: {"cases": 400, "expected": 400, "succeeded": 400, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 400

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/synthetic_relevance__nanobeir__NanoHotpotQA / test · binary_brier. Population 400 cases / 400 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"binary_brier":0.022710947594886247,"accuracy":0.912875,"positive_recall":0.8219718309859155,"specificity":0.9387951807228915,"precision":0.7929347826086957,"balanced_accuracy":0.8803835058544035,"f1":0.807192254495159,"true_positive":72.95,"false_positive":19.05,"false_negative":15.8,"true_negative":292.2,"positive_prevalence":0.221875,"majority_accuracy_baseline":0.778125,"constant_brier_baseline":0.12674023437499998,"brier_skill":0.8208071201155513,"baseline_adjusted_skill":0.760767011708807,"baseline_adjusted_score":0.760767011708807}. Counts: {"cases": 400, "expected": 400, "succeeded": 400, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 400

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/synthetic_relevance__nanobeir__NanoHotpotQA / test · normalized_expected_score_mae. Population 400 cases / 400 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"normalized_expected_score_mae":0.09327172372253573,"normalized_score_rmse":0.14013234166306046,"constant_score_baseline":0,"constant_mae_baseline":0.225625,"baseline_adjusted_skill":0.5866073186812821,"baseline_adjusted_score":0.5866073186812821}. Counts: {"cases": 400, "expected": 400, "succeeded": 400, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 400

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/synthetic_relevance__nanobeir__NanoMSMARCO / test · binary_brier. Population 400 cases / 400 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"binary_brier":0.03149015389442183,"accuracy":0.8554999999999999,"positive_recall":0.727788279773157,"specificity":0.9014276002719238,"precision":0.7264150943396226,"balanced_accuracy":0.8146079400225403,"f1":0.7271010387157696,"true_positive":77,"false_positive":29,"false_negative":28.8,"true_negative":265.2,"positive_prevalence":0.2645,"majority_accuracy_baseline":0.7355,"constant_brier_baseline":0.11362725,"brier_skill":0.7228644194555283,"baseline_adjusted_skill":0.6292158800450807,"baseline_adjusted_score":0.6292158800450807}. Counts: {"cases": 400, "expected": 400, "succeeded": 400, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 400

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/synthetic_relevance__nanobeir__NanoMSMARCO / test · normalized_expected_score_mae. Population 400 cases / 400 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"normalized_expected_score_mae":0.1177217332705277,"normalized_score_rmse":0.15927103862221628,"constant_score_baseline":0.25,"constant_mae_baseline":0.25125,"baseline_adjusted_skill":0.5314557879779991,"baseline_adjusted_score":0.5314557879779991}. Counts: {"cases": 400, "expected": 400, "succeeded": 400, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 400

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/synthetic_relevance__nanobeir__NanoNFCorpus / test · binary_brier. Population 400 cases / 400 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"binary_brier":0.05684565745441532,"accuracy":0.8275,"positive_recall":0.3302872062663186,"specificity":0.9452690166975881,"precision":0.5883720930232559,"balanced_accuracy":0.6377781114819534,"f1":0.42307692307692313,"true_positive":25.3,"false_positive":17.7,"false_negative":51.3,"true_negative":305.7,"positive_prevalence":0.19149999999999998,"majority_accuracy_baseline":0.8085,"constant_brier_baseline":0.09190274999999999,"brier_skill":0.3814585803535223,"baseline_adjusted_skill":0.2755562229639068,"baseline_adjusted_score":0.2755562229639068}. Counts: {"cases": 400, "expected": 400, "succeeded": 400, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 400

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/synthetic_relevance__nanobeir__NanoNFCorpus / test · normalized_expected_score_mae. Population 400 cases / 400 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"normalized_expected_score_mae":0.13691315379258326,"normalized_score_rmse":0.21560693806313136,"constant_score_baseline":0,"constant_mae_baseline":0.206875,"baseline_adjusted_skill":0.3381841508515613,"baseline_adjusted_score":0.3381841508515613}. Counts: {"cases": 400, "expected": 400, "succeeded": 400, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 400

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/synthetic_relevance__nanobeir__NanoNQ / test · binary_brier. Population 400 cases / 400 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"binary_brier":0.027620765479492406,"accuracy":0.9102499999999999,"positive_recall":0.7378048780487804,"specificity":0.9492642550582464,"precision":0.7669014084507043,"balanced_accuracy":0.8435345665535134,"f1":0.7520718232044199,"true_positive":54.45,"false_positive":16.55,"false_negative":19.35,"true_negative":309.65,"positive_prevalence":0.1845,"majority_accuracy_baseline":0.8155,"constant_brier_baseline":0.10922225,"brier_skill":0.7471141138413427,"baseline_adjusted_skill":0.6870691331070269,"baseline_adjusted_score":0.6870691331070269}. Counts: {"cases": 400, "expected": 400, "succeeded": 400, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 400

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/synthetic_relevance__nanobeir__NanoNQ / test · normalized_expected_score_mae. Population 400 cases / 400 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"normalized_expected_score_mae":0.11513610951873493,"normalized_score_rmse":0.17064070585219523,"constant_score_baseline":0,"constant_mae_baseline":0.189375,"baseline_adjusted_skill":0.39202054379545903,"baseline_adjusted_score":0.39202054379545903}. Counts: {"cases": 400, "expected": 400, "succeeded": 400, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 400

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/synthetic_relevance__nanobeir__NanoSCIDOCS / test · binary_brier. Population 400 cases / 400 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"binary_brier":0.04564165295053654,"accuracy":0.856,"positive_recall":0.40700636942675156,"specificity":0.9656298600311042,"precision":0.7430232558139535,"balanced_accuracy":0.6863181147289279,"f1":0.5259259259259259,"true_positive":31.95,"false_positive":11.049999999999999,"false_negative":46.55,"true_negative":310.45,"positive_prevalence":0.19625,"majority_accuracy_baseline":0.80375,"constant_brier_baseline":0.09878593749999999,"brier_skill":0.5379741883753794,"baseline_adjusted_skill":0.3726362294578558,"baseline_adjusted_score":0.3726362294578558}. Counts: {"cases": 400, "expected": 400, "succeeded": 400, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 400

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/synthetic_relevance__nanobeir__NanoSCIDOCS / test · normalized_expected_score_mae. Population 400 cases / 400 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"normalized_expected_score_mae":0.15063310979757402,"normalized_score_rmse":0.22071179619351253,"constant_score_baseline":0,"constant_mae_baseline":0.21125,"baseline_adjusted_skill":0.2869438589463952,"baseline_adjusted_score":0.2869438589463952}. Counts: {"cases": 400, "expected": 400, "succeeded": 400, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 400

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/synthetic_relevance__nanobeir__NanoTouche2020 / test · binary_brier. Population 392 cases / 392 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"binary_brier":0.0773978691264872,"accuracy":0.7104591836734694,"positive_recall":0.8840450771055752,"specificity":0.5794538943598926,"precision":0.6133744855967078,"balanced_accuracy":0.7317494857327339,"f1":0.7242468415937804,"true_positive":149.05,"false_positive":93.95,"false_negative":19.55,"true_negative":129.45,"positive_prevalence":0.4301020408163265,"majority_accuracy_baseline":0.5698979591836735,"constant_brier_baseline":0.12359641815910039,"brier_skill":0.37378550058905236,"baseline_adjusted_skill":0.4634989714654678,"baseline_adjusted_score":0.4634989714654678}. Counts: {"cases": 392, "expected": 392, "succeeded": 392, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 392

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/synthetic_relevance__nanobeir__NanoTouche2020 / test · normalized_expected_score_mae. Population 392 cases / 392 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"normalized_expected_score_mae":0.15068674908262586,"normalized_score_rmse":0.19715061793298813,"constant_score_baseline":0.5,"constant_mae_baseline":0.2755102040816326,"baseline_adjusted_skill":0.45306291073713567,"baseline_adjusted_score":0.45306291073713567}. Counts: {"cases": 392, "expected": 392, "succeeded": 392, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 392

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/synthetic_relevance__nanocoir / test · binary_brier. Population 800 cases / 800 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"binary_brier":0.023865450518140277,"accuracy":0.9255,"positive_recall":0.6864673485444532,"specificity":0.9706494278496062,"precision":0.8154205607476636,"balanced_accuracy":0.8285583881970298,"f1":0.7454079453225118,"true_positive":87.25,"false_positive":19.75,"false_negative":39.85,"true_negative":653.15,"positive_prevalence":0.158875,"majority_accuracy_baseline":0.841125,"constant_brier_baseline":0.09877748437499999,"brier_skill":0.7583917967829874,"baseline_adjusted_skill":0.6571167763940595,"baseline_adjusted_score":0.6571167763940595}. Counts: {"cases": 800, "expected": 800, "succeeded": 800, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 800

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/synthetic_relevance__nanocoir / test · normalized_expected_score_mae. Population 800 cases / 800 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"normalized_expected_score_mae":0.10901647546946584,"normalized_score_rmse":0.15715593423665505,"constant_score_baseline":0,"constant_mae_baseline":0.165,"baseline_adjusted_skill":0.3392940880638434,"baseline_adjusted_score":0.3392940880638434}. Counts: {"cases": 800, "expected": 800, "succeeded": 800, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 800

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/tallyqa-score / test · normalized_expected_score_mae. Population 300 cases / 300 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"normalized_expected_score_mae":0.035011315023547214,"normalized_score_rmse":0.06685042599346616,"constant_score_baseline":0.1111111111111111,"constant_mae_baseline":0.10629629629629629,"baseline_adjusted_skill":0.6706252593603572,"baseline_adjusted_score":0.6706252593603572}. Counts: {"cases": 300, "expected": 300, "succeeded": 300, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 300

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/temporal_nli / test · target_mass_at_prediction. Population 100 cases / 100 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Choice target mass is a fraction converted to percent; soft targets are not ordinary hard-label accuracy. Diagnostics: {"target_mass_at_prediction":0.98,"uniform_accuracy_baseline":0.33333333333333326,"fixed_answer_accuracy_baseline":0.38,"baseline_adjusted_skill":0.9677419354838709,"baseline_adjusted_score":0.9677419354838709}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/tracie / test · binary_brier. Population 100 cases / 100 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"binary_brier":0.1591218032795784,"accuracy":0.78,"positive_recall":0.66,"specificity":0.9,"precision":0.868421052631579,"balanced_accuracy":0.78,"f1":0.75,"true_positive":33,"false_positive":5,"false_negative":17,"true_negative":45,"positive_prevalence":0.5,"majority_accuracy_baseline":0.5,"constant_brier_baseline":0.25,"brier_skill":0.3635127868816864,"baseline_adjusted_skill":0.56,"baseline_adjusted_score":0.56}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/ud_ewt / test · target_mass_at_prediction. Population 100 cases / 1509 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Choice target mass is a fraction converted to percent; soft targets are not ordinary hard-label accuracy. Diagnostics: {"target_mass_at_prediction":0.42677269715043076,"uniform_accuracy_baseline":0.058823529411764705,"fixed_answer_accuracy_baseline":0.14910536779324055,"baseline_adjusted_skill":0.3263239875389408,"baseline_adjusted_score":0.3263239875389408}. Counts: {"cases": 100, "expected": 1509, "succeeded": 1509, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/visa-noul / test · binary_brier. Population 300 cases / 300 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"binary_brier":0.2708933940262088,"accuracy":0.57,"positive_recall":0.26436781609195403,"specificity":0.9920634920634921,"precision":0.9787234042553191,"balanced_accuracy":0.6282156540777231,"f1":0.416289592760181,"true_positive":46,"false_positive":1,"false_negative":128,"true_negative":125,"positive_prevalence":0.58,"majority_accuracy_baseline":0.58,"constant_brier_baseline":0.24359999999999998,"brier_skill":-0.11204184739823009,"baseline_adjusted_skill":0.2564313081554461,"baseline_adjusted_score":0.2564313081554461}. Counts: {"cases": 300, "expected": 300, "succeeded": 300, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 300

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/vsr-noul / test · binary_brier. Population 300 cases / 300 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"binary_brier":0.12673755680449936,"accuracy":0.8433333333333334,"positive_recall":0.910828025477707,"specificity":0.7692307692307693,"precision":0.8125,"balanced_accuracy":0.8400293973542381,"f1":0.8588588588588588,"true_positive":143,"false_positive":33,"false_negative":14,"true_negative":110,"positive_prevalence":0.5233333333333333,"majority_accuracy_baseline":0.5233333333333333,"constant_brier_baseline":0.24945555555555557,"brier_skill":0.4919433382742443,"baseline_adjusted_skill":0.6800587947084762,"baseline_adjusted_score":0.6800587947084762}. Counts: {"cases": 300, "expected": 300, "succeeded": 300, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 300

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/winobias / test · target_mass_at_prediction. Population 100 cases / 100 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Choice target mass is a fraction converted to percent; soft targets are not ordinary hard-label accuracy. Diagnostics: {"target_mass_at_prediction":0.98,"uniform_accuracy_baseline":0.5,"fixed_answer_accuracy_baseline":0.54,"baseline_adjusted_skill":0.9565217391304347,"baseline_adjusted_score":0.9565217391304347}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/winowhy / test · binary_brier. Population 100 cases / 100 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Diagnostics: {"binary_brier":0.13234170960315106,"accuracy":0.82,"positive_recall":0.84,"specificity":0.8,"precision":0.8076923076923077,"balanced_accuracy":0.8200000000000001,"f1":0.8235294117647058,"true_positive":42,"false_positive":10,"false_negative":8,"true_negative":40,"positive_prevalence":0.5,"majority_accuracy_baseline":0.5,"constant_brier_baseline":0.25,"brier_skill":0.47063316158739577,"baseline_adjusted_skill":0.6400000000000001,"baseline_adjusted_score":0.6400000000000001}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

S1MB datasets/wiqa / test · target_mass_at_prediction. Population 100 cases / 100 decisions.

Exact vllm-sr/d3; adapter d3; settings {"questions_per_call": 1}. Run vllm-sr__d3.

Published complete measured S1MB summary, checksum verified. Execution/submission status is not assumed independent. Choice target mass is a fraction converted to percent; soft targets are not ordinary hard-label accuracy. Diagnostics: {"target_mass_at_prediction":0.63,"uniform_accuracy_baseline":0.33333333333333326,"fixed_answer_accuracy_baseline":0.37,"baseline_adjusted_skill":0.41269841269841273,"baseline_adjusted_score":0.41269841269841273}. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Exact executed weight SHA is not published. Source non-embedding parameter counts do not replace official tensor totals and are not MoE active counts.

Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero.

Checked / reported: 2026-10-19 · Samples: 100

Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4 · Published aggregate observations; result reuse terms not separately specified

Accuracy under third-party joint-request scorer, distinct from original hosted d1/submitter-specific ECE conventions.

Exact named vllm-sr/d3; zero-shot HF Jobs, vLLM nightly /v1/systemone, --max-model-len 8192 --max-num-seqs 8; one request per case, 400 cases with all 5 questions each; hardware varies and latency is not ranked. Exact weights SHA and vLLM revision unspecified by source summary.

Third-party SaylorTwift HF Jobs public evaluation, 16 October 2026; accepted into publisher card 19–20 October. Test: 400 cases / 2000 typed decisions; all five questions with the state in one /v1/systemone request. Argmax accuracy counts ties in gold as correct; KL(gold||prediction) clips predictions at 1e-9; native multiclass Brier; ECE over 10 equal-width confidence bins. Same public scorer across these 10 model rows; not rerun by dataset owner. This scorer differs from the older hosted d1 row, so separate dataset IDs. Heterogeneous GPU latency excluded. Teacher-produced soft labels define agreement, not objective correctness. Model checkpoint revision and exact vLLM nightly build are not published in the summary; exact named HF checkpoint, no sibling transfers. Submission: https://huggingface.co/datasets/LocalLLaMA/typed-decisions/discussions/37

Third-party SaylorTwift HF Jobs public evaluation, 16 October 2026; accepted into publisher card 19–20 October. Test: 400 cases / 2000 typed decisions; all five questions with the state in one /v1/systemone request. Argmax accuracy counts ties in gold as correct; KL(gold||prediction) clips predictions at 1e-9; native multiclass Brier; ECE over 10 equal-width confidence bins. Same public scorer across these 10 model rows; not rerun by dataset owner. This scorer differs from the older hosted d1 row, so separate dataset IDs. Heterogeneous GPU latency excluded. Teacher-produced soft labels define agreement, not objective correctness. Model checkpoint revision and exact vLLM nightly build are not published in the summary; exact named HF checkpoint, no sibling transfers.

Checked / reported: 2026-10-20 · Evaluated: 2026-10-16 · Samples: 2000

Snapshot: 2026-10-19T22:54:08.000Z · Revision: d9eda7f31ba227333213a1787f2c78fdaa4090c8 · Apache-2.0 dataset/card; third-party published observations

KL from gold under third-party joint-request scorer, distinct from original hosted d1/submitter-specific ECE conventions.

Exact named vllm-sr/d3; zero-shot HF Jobs, vLLM nightly /v1/systemone, --max-model-len 8192 --max-num-seqs 8; one request per case, 400 cases with all 5 questions each; hardware varies and latency is not ranked. Exact weights SHA and vLLM revision unspecified by source summary.

Third-party SaylorTwift HF Jobs public evaluation, 16 October 2026; accepted into publisher card 19–20 October. Test: 400 cases / 2000 typed decisions; all five questions with the state in one /v1/systemone request. Argmax accuracy counts ties in gold as correct; KL(gold||prediction) clips predictions at 1e-9; native multiclass Brier; ECE over 10 equal-width confidence bins. Same public scorer across these 10 model rows; not rerun by dataset owner. This scorer differs from the older hosted d1 row, so separate dataset IDs. Heterogeneous GPU latency excluded. Teacher-produced soft labels define agreement, not objective correctness. Model checkpoint revision and exact vLLM nightly build are not published in the summary; exact named HF checkpoint, no sibling transfers. Submission: https://huggingface.co/datasets/LocalLLaMA/typed-decisions/discussions/37

Third-party SaylorTwift HF Jobs public evaluation, 16 October 2026; accepted into publisher card 19–20 October. Test: 400 cases / 2000 typed decisions; all five questions with the state in one /v1/systemone request. Argmax accuracy counts ties in gold as correct; KL(gold||prediction) clips predictions at 1e-9; native multiclass Brier; ECE over 10 equal-width confidence bins. Same public scorer across these 10 model rows; not rerun by dataset owner. This scorer differs from the older hosted d1 row, so separate dataset IDs. Heterogeneous GPU latency excluded. Teacher-produced soft labels define agreement, not objective correctness. Model checkpoint revision and exact vLLM nightly build are not published in the summary; exact named HF checkpoint, no sibling transfers.

Checked / reported: 2026-10-20 · Evaluated: 2026-10-16 · Samples: 2000

Snapshot: 2026-10-19T22:54:08.000Z · Revision: d9eda7f31ba227333213a1787f2c78fdaa4090c8 · Apache-2.0 dataset/card; third-party published observations

Brier under third-party joint-request scorer, distinct from original hosted d1/submitter-specific ECE conventions.

Exact named vllm-sr/d3; zero-shot HF Jobs, vLLM nightly /v1/systemone, --max-model-len 8192 --max-num-seqs 8; one request per case, 400 cases with all 5 questions each; hardware varies and latency is not ranked. Exact weights SHA and vLLM revision unspecified by source summary.

Third-party SaylorTwift HF Jobs public evaluation, 16 October 2026; accepted into publisher card 19–20 October. Test: 400 cases / 2000 typed decisions; all five questions with the state in one /v1/systemone request. Argmax accuracy counts ties in gold as correct; KL(gold||prediction) clips predictions at 1e-9; native multiclass Brier; ECE over 10 equal-width confidence bins. Same public scorer across these 10 model rows; not rerun by dataset owner. This scorer differs from the older hosted d1 row, so separate dataset IDs. Heterogeneous GPU latency excluded. Teacher-produced soft labels define agreement, not objective correctness. Model checkpoint revision and exact vLLM nightly build are not published in the summary; exact named HF checkpoint, no sibling transfers. Submission: https://huggingface.co/datasets/LocalLLaMA/typed-decisions/discussions/37

Third-party SaylorTwift HF Jobs public evaluation, 16 October 2026; accepted into publisher card 19–20 October. Test: 400 cases / 2000 typed decisions; all five questions with the state in one /v1/systemone request. Argmax accuracy counts ties in gold as correct; KL(gold||prediction) clips predictions at 1e-9; native multiclass Brier; ECE over 10 equal-width confidence bins. Same public scorer across these 10 model rows; not rerun by dataset owner. This scorer differs from the older hosted d1 row, so separate dataset IDs. Heterogeneous GPU latency excluded. Teacher-produced soft labels define agreement, not objective correctness. Model checkpoint revision and exact vLLM nightly build are not published in the summary; exact named HF checkpoint, no sibling transfers.

Checked / reported: 2026-10-20 · Evaluated: 2026-10-16 · Samples: 2000

Snapshot: 2026-10-19T22:54:08.000Z · Revision: d9eda7f31ba227333213a1787f2c78fdaa4090c8 · Apache-2.0 dataset/card; third-party published observations

ECE under third-party joint-request scorer, distinct from original hosted d1/submitter-specific ECE conventions.

Exact named vllm-sr/d3; zero-shot HF Jobs, vLLM nightly /v1/systemone, --max-model-len 8192 --max-num-seqs 8; one request per case, 400 cases with all 5 questions each; hardware varies and latency is not ranked. Exact weights SHA and vLLM revision unspecified by source summary.

Third-party SaylorTwift HF Jobs public evaluation, 16 October 2026; accepted into publisher card 19–20 October. Test: 400 cases / 2000 typed decisions; all five questions with the state in one /v1/systemone request. Argmax accuracy counts ties in gold as correct; KL(gold||prediction) clips predictions at 1e-9; native multiclass Brier; ECE over 10 equal-width confidence bins. Same public scorer across these 10 model rows; not rerun by dataset owner. This scorer differs from the older hosted d1 row, so separate dataset IDs. Heterogeneous GPU latency excluded. Teacher-produced soft labels define agreement, not objective correctness. Model checkpoint revision and exact vLLM nightly build are not published in the summary; exact named HF checkpoint, no sibling transfers. Submission: https://huggingface.co/datasets/LocalLLaMA/typed-decisions/discussions/37

Third-party SaylorTwift HF Jobs public evaluation, 16 October 2026; accepted into publisher card 19–20 October. Test: 400 cases / 2000 typed decisions; all five questions with the state in one /v1/systemone request. Argmax accuracy counts ties in gold as correct; KL(gold||prediction) clips predictions at 1e-9; native multiclass Brier; ECE over 10 equal-width confidence bins. Same public scorer across these 10 model rows; not rerun by dataset owner. This scorer differs from the older hosted d1 row, so separate dataset IDs. Heterogeneous GPU latency excluded. Teacher-produced soft labels define agreement, not objective correctness. Model checkpoint revision and exact vLLM nightly build are not published in the summary; exact named HF checkpoint, no sibling transfers.

Checked / reported: 2026-10-20 · Evaluated: 2026-10-16 · Samples: 2000

Snapshot: 2026-10-19T22:54:08.000Z · Revision: d9eda7f31ba227333213a1787f2c78fdaa4090c8 · Apache-2.0 dataset/card; third-party published observations