Examenos

BFCL formulated as typed decisions; published case exact accuracy for this board population.

Board identity d1-3b; author-submitted public run; private tests evaluated by the board. Public model-card serving path; actual revision and precision not otherwise established.

Raw 0.8937; {"requests": 1694, "answered": 1694, "unsupported": 0, "errors": 0}; coverage/error rules are source-defined.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: 1694

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

SGD/SGD-X formulated as typed decisions; published macro-F1 for this board population.

Board identity d1-3b; author-submitted public run; private tests evaluated by the board. Public model-card serving path; actual revision and precision not otherwise established.

Raw 0.692; {"requests": 2500, "answered": 2500, "unsupported": 0, "errors": 0}; coverage/error rules are source-defined.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: 2500

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

ContractNLI formulated as typed decisions; published macro-F1 for this board population.

Board identity d1-3b; author-submitted public run; private tests evaluated by the board. Public model-card serving path; actual revision and precision not otherwise established.

Raw 0.8057; {"requests": 123, "answered": 123, "unsupported": 0, "errors": 0}; coverage/error rules are source-defined.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: 123

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

ANLI formulated as typed decisions; published macro-F1 for this board population.

Board identity d1-3b; author-submitted public run; private tests evaluated by the board. Public model-card serving path; actual revision and precision not otherwise established.

Raw 0.6238; {"requests": 3200, "answered": 3200, "unsupported": 0, "errors": 0}; coverage/error rules are source-defined.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: 3200

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

ToolRet formulated as typed decisions; published nDCG@10 for this board population.

Board identity d1-3b; author-submitted public run; private tests evaluated by the board. Public model-card serving path; actual revision and precision not otherwise established.

Raw 0.6574; {"requests": 685, "answered": 685, "unsupported": 0, "errors": 0}; coverage/error rules are source-defined.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: 685

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

BPoMP formulated as typed decisions; published accuracy for this board population.

Board identity d1-3b; author-submitted public run; private tests evaluated by the board. Public model-card serving path; actual revision and precision not otherwise established.

Raw 0.8236; {"requests": 5000, "answered": 5000, "unsupported": 0, "errors": 0}; coverage/error rules are source-defined.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: 5000

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

Humicroedit formulated as typed decisions; published accuracy for this board population.

Board identity d1-3b; author-submitted public run; private tests evaluated by the board. Public model-card serving path; actual revision and precision not otherwise established.

Raw 0.6362; {"requests": 2628, "answered": 2628, "unsupported": 0, "errors": 0}; coverage/error rules are source-defined.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: 2628

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

POP909-CL formulated as typed decisions; published accuracy for this board population.

Board identity d1-3b; author-submitted public run; private tests evaluated by the board. Public model-card serving path; actual revision and precision not otherwise established.

Raw 0.253; {"requests": 2000, "answered": 2000, "unsupported": 0, "errors": 0}; coverage/error rules are source-defined.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: 2000

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

cfcolor formulated as typed decisions; published accuracy for this board population.

Board identity d1-3b; author-submitted public run; private tests evaluated by the board. Public model-card serving path; actual revision and precision not otherwise established.

Raw 0.6204; {"requests": 5000, "answered": 5000, "unsupported": 0, "errors": 0}; coverage/error rules are source-defined.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: 5000

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

MMLU formulated as typed decisions; published accuracy for this board population.

Board identity d1-3b; author-submitted public run; private tests evaluated by the board. Public model-card serving path; actual revision and precision not otherwise established.

Raw 0.6662; {"requests": 14033, "answered": 14033, "unsupported": 0, "errors": 0}; coverage/error rules are source-defined.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: 14033

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

GPQA Diamond formulated as typed decisions; published accuracy for this board population.

Board identity d1-3b; author-submitted public run; private tests evaluated by the board. Public model-card serving path; actual revision and precision not otherwise established.

Raw 0.2857; {"requests": 196, "answered": 196, "unsupported": 0, "errors": 0}; coverage/error rules are source-defined.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: 196

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

ARC-Easy formulated as typed decisions; published accuracy for this board population.

Board identity d1-3b; author-submitted public run; private tests evaluated by the board. Public model-card serving path; actual revision and precision not otherwise established.

Raw 0.9537; {"requests": 2376, "answered": 2376, "unsupported": 0, "errors": 0}; coverage/error rules are source-defined.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: 2376

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

ARC-Challenge formulated as typed decisions; published accuracy for this board population.

Board identity d1-3b; author-submitted public run; private tests evaluated by the board. Public model-card serving path; actual revision and precision not otherwise established.

Raw 0.8797; {"requests": 1172, "answered": 1172, "unsupported": 0, "errors": 0}; coverage/error rules are source-defined.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: 1172

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

WinoGrande formulated as typed decisions; published accuracy for this board population.

Board identity d1-3b; author-submitted public run; private tests evaluated by the board. Public model-card serving path; actual revision and precision not otherwise established.

Raw 0.7774; {"requests": 1267, "answered": 1267, "unsupported": 0, "errors": 0}; coverage/error rules are source-defined.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: 1267

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

HellaSwag formulated as typed decisions; published accuracy for this board population.

Board identity d1-3b; author-submitted public run; private tests evaluated by the board. Public model-card serving path; actual revision and precision not otherwise established.

Raw 0.8922; {"requests": 10042, "answered": 10042, "unsupported": 0, "errors": 0}; coverage/error rules are source-defined.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: 10042

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

API-Bank formulated as typed decisions; published accuracy for this board population.

Board identity d1-3b; author-submitted public run; private tests evaluated by the board. Public model-card serving path; actual revision and precision not otherwise established.

Raw 0.7815; {"requests": 508, "answered": 508, "unsupported": 0, "errors": 0}; coverage/error rules are source-defined.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: 508

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

GSM8K formulated as typed decisions; published accuracy for this board population.

Board identity d1-3b; author-submitted public run; private tests evaluated by the board. Public model-card serving path; actual revision and precision not otherwise established.

Raw 0.3268; {"requests": 2638, "answered": 2638, "unsupported": 0, "errors": 0}; coverage/error rules are source-defined.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: 2638

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

ChessBench formulated as typed decisions; published accuracy for this board population.

Board identity d1-3b; author-submitted public run; private tests evaluated by the board. Public model-card serving path; actual revision and precision not otherwise established.

Raw 0.1578; {"requests": 5000, "answered": 5000, "unsupported": 0, "errors": 0}; coverage/error rules are source-defined.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: 5000

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

MuSR formulated as typed decisions; published accuracy for this board population.

Board identity d1-3b; author-submitted public run; private tests evaluated by the board. Public model-card serving path; actual revision and precision not otherwise established.

Raw 0.5678; {"requests": 752, "answered": 752, "unsupported": 0, "errors": 0}; coverage/error rules are source-defined.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: 752

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

SATA-Bench formulated as typed decisions; published case exact accuracy for this board population.

Board identity d1-3b; author-submitted public run; private tests evaluated by the board. Public model-card serving path; actual revision and precision not otherwise established.

Raw 0.3133; {"requests": 1650, "answered": 1650, "unsupported": 0, "errors": 0}; coverage/error rules are source-defined.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: 1650

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

BRIGHT formulated as typed decisions; published nDCG@10 for this board population.

Board identity d1-3b; author-submitted public run; private tests evaluated by the board. Public model-card serving path; actual revision and precision not otherwise established.

Raw 0.4057; {"requests": 220, "answered": 220, "unsupported": 0, "errors": 0}; coverage/error rules are source-defined.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: 220

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

Amazon ESCI formulated as typed decisions; published macro-F1 for this board population.

Board identity d1-3b; author-submitted public run; private tests evaluated by the board. Public model-card serving path; actual revision and precision not otherwise established.

Raw 0.5124; {"requests": 5000, "answered": 5000, "unsupported": 0, "errors": 0}; coverage/error rules are source-defined.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: 5000

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

ACOS formulated as typed decisions; published per-review F1 for this board population.

Board identity d1-3b; author-submitted public run; private tests evaluated by the board. Public model-card serving path; actual revision and precision not otherwise established.

Raw 0.377; {"requests": 1565, "answered": 1565, "unsupported": 0, "errors": 0}; coverage/error rules are source-defined.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: 1565

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

FinEntity formulated as typed decisions; published macro-F1 for this board population.

Board identity d1-3b; author-submitted public run; private tests evaluated by the board. Public model-card serving path; actual revision and precision not otherwise established.

Raw 0.8804; {"requests": 979, "answered": 979, "unsupported": 0, "errors": 0}; coverage/error rules are source-defined.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: 979

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

BANKING77 formulated as typed decisions; published macro-F1 for this board population.

Board identity d1-3b; author-submitted public run; private tests evaluated by the board. Public model-card serving path; actual revision and precision not otherwise established.

Raw 0.8797; {"requests": 3080, "answered": 3080, "unsupported": 0, "errors": 0}; coverage/error rules are source-defined.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: 3080

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

iSarcasmEval formulated as typed decisions; published Sarcasm F1 · track A, English for this board population.

Board identity d1-3b; author-submitted public run; private tests evaluated by the board. Public model-card serving path; actual revision and precision not otherwise established.

Raw 0.3597; {"requests": 4600, "answered": 4600, "unsupported": 0, "errors": 0}; coverage/error rules are source-defined.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: 4600

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

VAST formulated as typed decisions; published macro-F1 for this board population.

Board identity d1-3b; author-submitted public run; private tests evaluated by the board. Public model-card serving path; actual revision and precision not otherwise established.

Raw 0.7696; {"requests": 3006, "answered": 3006, "unsupported": 0, "errors": 0}; coverage/error rules are source-defined.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: 3006

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

NLI4CT formulated as typed decisions; published macro-F1 for this board population.

Board identity d1-3b; author-submitted public run; private tests evaluated by the board. Public model-card serving path; actual revision and precision not otherwise established.

Raw 0.7412; {"requests": 5500, "answered": 5500, "unsupported": 0, "errors": 0}; coverage/error rules are source-defined.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: 5500

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

CRUXEval formulated as typed decisions; published accuracy for this board population.

Board identity d1-3b; author-submitted public run; private tests evaluated by the board. Public model-card serving path; actual revision and precision not otherwise established.

Raw 0.3105; {"requests": 570, "answered": 570, "unsupported": 0, "errors": 0}; coverage/error rules are source-defined.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: 570

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

CLadder formulated as typed decisions; published accuracy for this board population.

Board identity d1-3b; author-submitted public run; private tests evaluated by the board. Public model-card serving path; actual revision and precision not otherwise established.

Raw 0.5658; {"requests": 5000, "answered": 5000, "unsupported": 0, "errors": 0}; coverage/error rules are source-defined.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: 5000

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

HLE formulated as typed decisions; published accuracy for this board population.

Board identity d1-3b; author-submitted public run; private tests evaluated by the board. Public model-card serving path; actual revision and precision not otherwise established.

Raw 0.1417; {"requests": 501, "answered": 501, "unsupported": 0, "errors": 0}; coverage/error rules are source-defined.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: 501

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

CLINC150+OOS formulated as typed decisions; published macro-F1 for this board population.

Board identity d1-3b; author-submitted public run; private tests evaluated by the board. Public model-card serving path; actual revision and precision not otherwise established.

Raw 0.9119; {"requests": 5500, "answered": 5500, "unsupported": 0, "errors": 0}; coverage/error rules are source-defined.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: 5500

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

Habermas Machine formulated as typed decisions; published accuracy for this board population.

Board identity d1-3b; author-submitted public run; private tests evaluated by the board. Public model-card serving path; actual revision and precision not otherwise established.

Raw 0.5877; {"requests": 1676, "answered": 1676, "unsupported": 0, "errors": 0}; coverage/error rules are source-defined.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: 1676

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

PhishNChips phishing decisions formulated as typed decisions; published accuracy for this board population.

Board identity d1-3b; author-submitted public run; private tests evaluated by the board. Public model-card serving path; actual revision and precision not otherwise established.

Raw 0.5385; {"requests": 2000, "answered": 2000, "unsupported": 0, "errors": 0}; coverage/error rules are source-defined.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: 2000

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

MMLU-Pro formulated as typed decisions; published accuracy for this board population.

Board identity d1-3b; author-submitted public run; private tests evaluated by the board. Public model-card serving path; actual revision and precision not otherwise established.

Raw 0.4485; {"requests": 12032, "answered": 12032, "unsupported": 0, "errors": 0}; coverage/error rules are source-defined.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: 12032

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

BBH formulated as typed decisions; published accuracy for this board population.

Board identity d1-3b; author-submitted public run; private tests evaluated by the board. Public model-card serving path; actual revision and precision not otherwise established.

Raw 0.724; {"requests": 5507, "answered": 5507, "unsupported": 0, "errors": 0}; coverage/error rules are source-defined.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: 5507

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

RAGTruth formulated as typed decisions; published F1 on hallucinated class for this board population.

Board identity d1-3b; author-submitted public run; private tests evaluated by the board. Public model-card serving path; actual revision and precision not otherwise established.

Raw 0.7757; {"requests": 2700, "answered": 2700, "unsupported": 0, "errors": 0}; coverage/error rules are source-defined.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: 2700

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

RouterBench formulated as typed decisions; published selected quality (quality objective) for this board population.

Board identity d1-3b; author-submitted public run; private tests evaluated by the board. Public model-card serving path; actual revision and precision not otherwise established.

Raw 0.7946; {"requests": 10000, "answered": 10000, "unsupported": 0, "errors": 0}; coverage/error rules are source-defined.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: 10000

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

HoVer formulated as typed decisions; published accuracy for this board population.

Board identity d1-3b; author-submitted public run; private tests evaluated by the board. Public model-card serving path; actual revision and precision not otherwise established.

Raw 0.7408; {"requests": 4000, "answered": 4000, "unsupported": 0, "errors": 0}; coverage/error rules are source-defined.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: 4000

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

When2Call formulated as typed decisions; published accuracy for this board population.

Board identity d1-3b; author-submitted public run; private tests evaluated by the board. Public model-card serving path; actual revision and precision not otherwise established.

Raw 0.8658; {"requests": 3652, "answered": 3652, "unsupported": 0, "errors": 0}; coverage/error rules are source-defined.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: 3652

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

New Yorker formulated as typed decisions; published accuracy for this board population.

Board identity d1-3b; author-submitted public run; private tests evaluated by the board. Public model-card serving path; actual revision and precision not otherwise established.

Raw 0.6402; {"requests": 528, "answered": 528, "unsupported": 0, "errors": 0}; coverage/error rules are source-defined.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: 528

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

Home appliance simulator formulated as typed decisions; published case exact accuracy for this board population.

Board identity d1-3b; author-submitted public run; private tests evaluated by the board. Public model-card serving path; actual revision and precision not otherwise established.

Raw 0.6364; {"requests": 88, "answered": 88, "unsupported": 0, "errors": 0}; coverage/error rules are source-defined.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: 88

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

Calibration on the source-defined text sample; confidence alone is not a quality ranking.

Board identity d1-3b; author-submitted public run; private tests evaluated by the board. Public model-card serving path; actual revision and precision not otherwise established.

Raw acc: 0.7024; 31 benchmarks. ECE uses the publisher bins and weighting; native Brier convention stays distinct from other studies.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported. Uses roughly one sixth of requests with discrete correct answers; benchmarks receive equal weight. ToolRet, BRIGHT and RouterBench are excluded. Known training exposure is penalized.

Checked / reported: 2026-10-17 · Samples: 34411

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

Calibration on the source-defined text sample; confidence alone is not a quality ranking.

Board identity d1-3b; author-submitted public run; private tests evaluated by the board. Public model-card serving path; actual revision and precision not otherwise established.

Raw brier: 0.3902; 31 benchmarks. ECE uses the publisher bins and weighting; native Brier convention stays distinct from other studies.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported. Uses roughly one sixth of requests with discrete correct answers; benchmarks receive equal weight. ToolRet, BRIGHT and RouterBench are excluded. Known training exposure is penalized.

Checked / reported: 2026-10-17 · Samples: 34411

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

Calibration on the source-defined text sample; confidence alone is not a quality ranking.

Board identity d1-3b; author-submitted public run; private tests evaluated by the board. Public model-card serving path; actual revision and precision not otherwise established.

Raw conf: 0.7492; 31 benchmarks. ECE uses the publisher bins and weighting; native Brier convention stays distinct from other studies.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported. Uses roughly one sixth of requests with discrete correct answers; benchmarks receive equal weight. ToolRet, BRIGHT and RouterBench are excluded. Known training exposure is penalized.

Checked / reported: 2026-10-17 · Samples: 34411

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

Calibration on the source-defined text sample; confidence alone is not a quality ranking.

Board identity d1-3b; author-submitted public run; private tests evaluated by the board. Public model-card serving path; actual revision and precision not otherwise established.

Raw ece: 0.0468; 31 benchmarks. ECE uses the publisher bins and weighting; native Brier convention stays distinct from other studies.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported. Uses roughly one sixth of requests with discrete correct answers; benchmarks receive equal weight. ToolRet, BRIGHT and RouterBench are excluded. Known training exposure is penalized.

Checked / reported: 2026-10-17 · Samples: 34411

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

Published Median latency for the board setup; hosted network and local inference are separate.

Board identity d1-3b; author-submitted public run; private tests evaluated by the board. Public model-card serving path; actual revision and precision not otherwise established.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported. Local GPU/runtime can differ between submitted and evaluator runs; not isolated architecture speed.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

Published p95 latency for the board setup; hosted network and local inference are separate.

Board identity d1-3b; author-submitted public run; private tests evaluated by the board. Public model-card serving path; actual revision and precision not otherwise established.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported. Local GPU/runtime can differ between submitted and evaluator runs; not isolated architecture speed.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

Text Decision Index 0.3.2 · Private new-domain score

Board identity d1-3b; author-submitted public run; private tests evaluated by the board. Public model-card serving path; actual revision and precision not otherwise established.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

Text Decision Index 0.3.2 · Public score

Board identity d1-3b; author-submitted public run; private tests evaluated by the board. Public model-card serving path; actual revision and precision not otherwise established.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

Text Decision Index 0.3.2 · Private same-skill score

Board identity d1-3b; author-submitted public run; private tests evaluated by the board. Public model-card serving path; actual revision and precision not otherwise established.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

Text Decision Index 0.3.2 · Full score

Board identity d1-3b; author-submitted public run; private tests evaluated by the board. Public model-card serving path; actual revision and precision not otherwise established.

Community Decision Index 0.3.2; distinct from AA. Full = 20% public + 50% private same-skills + 30% private new-domains on publisher-normalized scales. Public suite is chance-corrected and coverage-adjusted; unanswered counts wrong. Some public runs are author-submitted and private runs evaluator-run; each record preserves that status. Small differences can be tied by the publisher. Snapshot date is not execution date. No underlying private test text is imported.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-16T16:39:08Z · Revision: 0ddf6758073eb77ae449a3541f57286d7f39d93f · Published observations; result reuse terms not specified

Source-specific published open d1-3B measurement.

Released open d1-3B checkpoint. All 23,900 English rows

All 23,900 English rows

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-07 · Revision: 051bcc464b01b9f92942b364d9586b0ef5912432 · Published observations; result reuse terms not specified

Source-specific published open d1-3B measurement.

Released open d1-3B checkpoint. Dev split

Dev split

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-07 · Revision: 051bcc464b01b9f92942b364d9586b0ef5912432 · Published observations; result reuse terms not specified

Source-specific published open d1-3B measurement.

Released open d1-3B checkpoint. Published vendor claim; request distribution not fully specified

Published vendor claim; request distribution not fully specified

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-07 · Revision: 051bcc464b01b9f92942b364d9586b0ef5912432 · Published observations; result reuse terms not specified

Source-specific published open d1-3B measurement.

Released open d1-3B checkpoint. Published vendor claim; request distribution not fully specified

Published vendor claim; request distribution not fully specified

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-07 · Revision: 051bcc464b01b9f92942b364d9586b0ef5912432 · Published observations; result reuse terms not specified

Source-specific published open d1-3B measurement.

Released open d1-3B checkpoint. Published vendor claim; request distribution not fully specified

Published vendor claim; request distribution not fully specified

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-07 · Revision: 051bcc464b01b9f92942b364d9586b0ef5912432 · Published observations; result reuse terms not specified

Source-specific published open d1-3B measurement.

Released open d1-3B checkpoint. Vendor removes images for the same image questions; separate ablation

Vendor removes images for the same image questions; separate ablation

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-07 · Revision: 051bcc464b01b9f92942b364d9586b0ef5912432 · Published observations; result reuse terms not specified

TypeSafe’s evolving private internal measurement; its own warning says likely biased toward its models.

TypeSafe release figure; exact named checkpoint. Internal suite/hardware and detailed calibration definition unpublished.

Figure-only printed values manually inspected. No stable suite or metric formula is published. Not the AA or HF Decision Index, and not comparable to other ECE definitions.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-14 · Revision: Launch figures 2026-10-14 · Published observations; result reuse terms not specified

TypeSafe’s evolving private internal measurement; its own warning says likely biased toward its models.

TypeSafe release figure; exact named checkpoint. Internal suite/hardware and detailed calibration definition unpublished.

Figure-only printed values manually inspected. No stable suite or metric formula is published. Not the AA or HF Decision Index, and not comparable to other ECE definitions.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-14 · Revision: Launch figures 2026-10-14 · Published observations; result reuse terms not specified

Benchmarks as decisions · BoolQ as reported by Liquid AI

Card LiquidAI/d1-3B at 051bcc464b01b9f92942b364d9586b0ef5912432; table Benchmarks as decisions.

Published table label: BoolQ.

Vendor-published observation, distinct from independent and other vendor harnesses. Source-specific open-d1 public Decision Index 0.2.1 and decision adaptations; model-name sizes are not benchmark scores. Means refer only to each card’s named task panel.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-07 · Revision: 051bcc464b01b9f92942b364d9586b0ef5912432 · Published observations; result reuse terms not specified

Benchmarks as decisions · Civil Comments as reported by Liquid AI

Card LiquidAI/d1-3B at 051bcc464b01b9f92942b364d9586b0ef5912432; table Benchmarks as decisions.

Published table label: Civil Comments.

Vendor-published observation, distinct from independent and other vendor harnesses. Source-specific open-d1 public Decision Index 0.2.1 and decision adaptations; model-name sizes are not benchmark scores. Means refer only to each card’s named task panel.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-07 · Revision: 051bcc464b01b9f92942b364d9586b0ef5912432 · Published observations; result reuse terms not specified

Benchmarks as decisions · HelpSteer2 as reported by Liquid AI

Card LiquidAI/d1-3B at 051bcc464b01b9f92942b364d9586b0ef5912432; table Benchmarks as decisions.

Published table label: HelpSteer2.

Vendor-published observation, distinct from independent and other vendor harnesses. Source-specific open-d1 public Decision Index 0.2.1 and decision adaptations; model-name sizes are not benchmark scores. Means refer only to each card’s named task panel.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-07 · Revision: 051bcc464b01b9f92942b364d9586b0ef5912432 · Published observations; result reuse terms not specified

Benchmarks as decisions · MASSIVE intent as reported by Liquid AI

Card LiquidAI/d1-3B at 051bcc464b01b9f92942b364d9586b0ef5912432; table Benchmarks as decisions.

Published table label: MASSIVE intent.

Vendor-published observation, distinct from independent and other vendor harnesses. Source-specific open-d1 public Decision Index 0.2.1 and decision adaptations; model-name sizes are not benchmark scores. Means refer only to each card’s named task panel.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-07 · Revision: 051bcc464b01b9f92942b364d9586b0ef5912432 · Published observations; result reuse terms not specified

Benchmarks as decisions · Mean as reported by Liquid AI

Card LiquidAI/d1-3B at 051bcc464b01b9f92942b364d9586b0ef5912432; table Benchmarks as decisions.

Published table label: Mean.

Vendor-published observation, distinct from independent and other vendor harnesses. Source-specific open-d1 public Decision Index 0.2.1 and decision adaptations; model-name sizes are not benchmark scores. Means refer only to each card’s named task panel.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-07 · Revision: 051bcc464b01b9f92942b364d9586b0ef5912432 · Published observations; result reuse terms not specified

Benchmarks as decisions · PAWS-X as reported by Liquid AI

Card LiquidAI/d1-3B at 051bcc464b01b9f92942b364d9586b0ef5912432; table Benchmarks as decisions.

Published table label: PAWS-X.

Vendor-published observation, distinct from independent and other vendor harnesses. Source-specific open-d1 public Decision Index 0.2.1 and decision adaptations; model-name sizes are not benchmark scores. Means refer only to each card’s named task panel.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-07 · Revision: 051bcc464b01b9f92942b364d9586b0ef5912432 · Published observations; result reuse terms not specified

Benchmarks as decisions · PubMedQA as reported by Liquid AI

Card LiquidAI/d1-3B at 051bcc464b01b9f92942b364d9586b0ef5912432; table Benchmarks as decisions.

Published table label: PubMedQA.

Vendor-published observation, distinct from independent and other vendor harnesses. Source-specific open-d1 public Decision Index 0.2.1 and decision adaptations; model-name sizes are not benchmark scores. Means refer only to each card’s named task panel.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-07 · Revision: 051bcc464b01b9f92942b364d9586b0ef5912432 · Published observations; result reuse terms not specified

Benchmarks as decisions · SQuAD 2.0 as reported by Liquid AI

Card LiquidAI/d1-3B at 051bcc464b01b9f92942b364d9586b0ef5912432; table Benchmarks as decisions.

Published table label: SQuAD 2.0.

Vendor-published observation, distinct from independent and other vendor harnesses. Source-specific open-d1 public Decision Index 0.2.1 and decision adaptations; model-name sizes are not benchmark scores. Means refer only to each card’s named task panel.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-07 · Revision: 051bcc464b01b9f92942b364d9586b0ef5912432 · Published observations; result reuse terms not specified

Benchmarks as decisions · XNLI as reported by Liquid AI

Card LiquidAI/d1-3B at 051bcc464b01b9f92942b364d9586b0ef5912432; table Benchmarks as decisions.

Published table label: XNLI.

Vendor-published observation, distinct from independent and other vendor harnesses. Source-specific open-d1 public Decision Index 0.2.1 and decision adaptations; model-name sizes are not benchmark scores. Means refer only to each card’s named task panel.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-07 · Revision: 051bcc464b01b9f92942b364d9586b0ef5912432 · Published observations; result reuse terms not specified

Benchmarks as decisions · BoolQ as reported by Liquid AI

Card LiquidAI/d1-omni-600M at 02b55d7076f15129e59ab3f94783f32c4b088674; table Benchmarks as decisions.

Published table label: BoolQ.

Vendor-published observation, distinct from independent and other vendor harnesses. Source-specific open-d1 public Decision Index 0.2.1 and decision adaptations; model-name sizes are not benchmark scores. Means refer only to each card’s named task panel.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-07 · Revision: 02b55d7076f15129e59ab3f94783f32c4b088674 · Published observations; result reuse terms not specified

Benchmarks as decisions · Civil Comments as reported by Liquid AI

Card LiquidAI/d1-omni-600M at 02b55d7076f15129e59ab3f94783f32c4b088674; table Benchmarks as decisions.

Published table label: Civil Comments.

Vendor-published observation, distinct from independent and other vendor harnesses. Source-specific open-d1 public Decision Index 0.2.1 and decision adaptations; model-name sizes are not benchmark scores. Means refer only to each card’s named task panel.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-07 · Revision: 02b55d7076f15129e59ab3f94783f32c4b088674 · Published observations; result reuse terms not specified

Benchmarks as decisions · MASSIVE intent as reported by Liquid AI

Card LiquidAI/d1-omni-600M at 02b55d7076f15129e59ab3f94783f32c4b088674; table Benchmarks as decisions.

Published table label: MASSIVE intent.

Vendor-published observation, distinct from independent and other vendor harnesses. Source-specific open-d1 public Decision Index 0.2.1 and decision adaptations; model-name sizes are not benchmark scores. Means refer only to each card’s named task panel.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-07 · Revision: 02b55d7076f15129e59ab3f94783f32c4b088674 · Published observations; result reuse terms not specified

Benchmarks as decisions · Mean as reported by Liquid AI

Card LiquidAI/d1-omni-600M at 02b55d7076f15129e59ab3f94783f32c4b088674; table Benchmarks as decisions.

Published table label: Mean.

Vendor-published observation, distinct from independent and other vendor harnesses. Source-specific open-d1 public Decision Index 0.2.1 and decision adaptations; model-name sizes are not benchmark scores. Means refer only to each card’s named task panel.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-07 · Revision: 02b55d7076f15129e59ab3f94783f32c4b088674 · Published observations; result reuse terms not specified

Benchmarks as decisions · PAWS-X as reported by Liquid AI

Card LiquidAI/d1-omni-600M at 02b55d7076f15129e59ab3f94783f32c4b088674; table Benchmarks as decisions.

Published table label: PAWS-X.

Vendor-published observation, distinct from independent and other vendor harnesses. Source-specific open-d1 public Decision Index 0.2.1 and decision adaptations; model-name sizes are not benchmark scores. Means refer only to each card’s named task panel.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-07 · Revision: 02b55d7076f15129e59ab3f94783f32c4b088674 · Published observations; result reuse terms not specified

Benchmarks as decisions · PubMedQA as reported by Liquid AI

Card LiquidAI/d1-omni-600M at 02b55d7076f15129e59ab3f94783f32c4b088674; table Benchmarks as decisions.

Published table label: PubMedQA.

Vendor-published observation, distinct from independent and other vendor harnesses. Source-specific open-d1 public Decision Index 0.2.1 and decision adaptations; model-name sizes are not benchmark scores. Means refer only to each card’s named task panel.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-07 · Revision: 02b55d7076f15129e59ab3f94783f32c4b088674 · Published observations; result reuse terms not specified

Benchmarks as decisions · SQuAD 2.0 as reported by Liquid AI

Card LiquidAI/d1-omni-600M at 02b55d7076f15129e59ab3f94783f32c4b088674; table Benchmarks as decisions.

Published table label: SQuAD 2.0.

Vendor-published observation, distinct from independent and other vendor harnesses. Source-specific open-d1 public Decision Index 0.2.1 and decision adaptations; model-name sizes are not benchmark scores. Means refer only to each card’s named task panel.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-07 · Revision: 02b55d7076f15129e59ab3f94783f32c4b088674 · Published observations; result reuse terms not specified

Benchmarks as decisions · XNLI as reported by Liquid AI

Card LiquidAI/d1-omni-600M at 02b55d7076f15129e59ab3f94783f32c4b088674; table Benchmarks as decisions.

Published table label: XNLI.

Vendor-published observation, distinct from independent and other vendor harnesses. Source-specific open-d1 public Decision Index 0.2.1 and decision adaptations; model-name sizes are not benchmark scores. Means refer only to each card’s named task panel.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-07 · Revision: 02b55d7076f15129e59ab3f94783f32c4b088674 · Published observations; result reuse terms not specified

Decision Index 0.2.1 · Arts as reported by Liquid AI

Card LiquidAI/d1-3B at 051bcc464b01b9f92942b364d9586b0ef5912432; table Decision Index 0.2.1.

Published table label: Arts.

Vendor-published observation, distinct from independent and other vendor harnesses. Source-specific open-d1 public Decision Index 0.2.1 and decision adaptations; model-name sizes are not benchmark scores. Means refer only to each card’s named task panel.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-07 · Revision: 051bcc464b01b9f92942b364d9586b0ef5912432 · Published observations; result reuse terms not specified

Decision Index 0.2.1 · Decision Index as reported by Liquid AI

Card LiquidAI/d1-3B at 051bcc464b01b9f92942b364d9586b0ef5912432; table Decision Index 0.2.1.

Published table label: Decision Index.

Vendor-published observation, distinct from independent and other vendor harnesses. Source-specific open-d1 public Decision Index 0.2.1 and decision adaptations; model-name sizes are not benchmark scores. Means refer only to each card’s named task panel.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-07 · Revision: 051bcc464b01b9f92942b364d9586b0ef5912432 · Published observations; result reuse terms not specified

Decision Index 0.2.1 · Knowledge as reported by Liquid AI

Card LiquidAI/d1-3B at 051bcc464b01b9f92942b364d9586b0ef5912432; table Decision Index 0.2.1.

Published table label: Knowledge.

Vendor-published observation, distinct from independent and other vendor harnesses. Source-specific open-d1 public Decision Index 0.2.1 and decision adaptations; model-name sizes are not benchmark scores. Means refer only to each card’s named task panel.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-07 · Revision: 051bcc464b01b9f92942b364d9586b0ef5912432 · Published observations; result reuse terms not specified

Decision Index 0.2.1 · Language as reported by Liquid AI

Card LiquidAI/d1-3B at 051bcc464b01b9f92942b364d9586b0ef5912432; table Decision Index 0.2.1.

Published table label: Language.

Vendor-published observation, distinct from independent and other vendor harnesses. Source-specific open-d1 public Decision Index 0.2.1 and decision adaptations; model-name sizes are not benchmark scores. Means refer only to each card’s named task panel.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-07 · Revision: 051bcc464b01b9f92942b364d9586b0ef5912432 · Published observations; result reuse terms not specified

Decision Index 0.2.1 · Retrieval as reported by Liquid AI

Card LiquidAI/d1-3B at 051bcc464b01b9f92942b364d9586b0ef5912432; table Decision Index 0.2.1.

Published table label: Retrieval.

Vendor-published observation, distinct from independent and other vendor harnesses. Source-specific open-d1 public Decision Index 0.2.1 and decision adaptations; model-name sizes are not benchmark scores. Means refer only to each card’s named task panel.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-07 · Revision: 051bcc464b01b9f92942b364d9586b0ef5912432 · Published observations; result reuse terms not specified

Decision Index 0.2.1 · Tools as reported by Liquid AI

Card LiquidAI/d1-3B at 051bcc464b01b9f92942b364d9586b0ef5912432; table Decision Index 0.2.1.

Published table label: Tools.

Vendor-published observation, distinct from independent and other vendor harnesses. Source-specific open-d1 public Decision Index 0.2.1 and decision adaptations; model-name sizes are not benchmark scores. Means refer only to each card’s named task panel.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-07 · Revision: 051bcc464b01b9f92942b364d9586b0ef5912432 · Published observations; result reuse terms not specified

Vision · AI2D as reported by Liquid AI

Card LiquidAI/d1-3B at 051bcc464b01b9f92942b364d9586b0ef5912432; table Vision.

Published table label: AI2D.

Vendor-published observation, distinct from independent and other vendor harnesses. Source-specific open-d1 public Decision Index 0.2.1 and decision adaptations; model-name sizes are not benchmark scores. Means refer only to each card’s named task panel.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-07 · Revision: 051bcc464b01b9f92942b364d9586b0ef5912432 · Published observations; result reuse terms not specified

Vision · BLINK as reported by Liquid AI

Card LiquidAI/d1-3B at 051bcc464b01b9f92942b364d9586b0ef5912432; table Vision.

Published table label: BLINK.

Vendor-published observation, distinct from independent and other vendor harnesses. Source-specific open-d1 public Decision Index 0.2.1 and decision adaptations; model-name sizes are not benchmark scores. Means refer only to each card’s named task panel.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-07 · Revision: 051bcc464b01b9f92942b364d9586b0ef5912432 · Published observations; result reuse terms not specified

Vision · CV-Bench as reported by Liquid AI

Card LiquidAI/d1-3B at 051bcc464b01b9f92942b364d9586b0ef5912432; table Vision.

Published table label: CV-Bench.

Vendor-published observation, distinct from independent and other vendor harnesses. Source-specific open-d1 public Decision Index 0.2.1 and decision adaptations; model-name sizes are not benchmark scores. Means refer only to each card’s named task panel.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-07 · Revision: 051bcc464b01b9f92942b364d9586b0ef5912432 · Published observations; result reuse terms not specified

Vision · HallusionBench as reported by Liquid AI

Card LiquidAI/d1-3B at 051bcc464b01b9f92942b364d9586b0ef5912432; table Vision.

Published table label: HallusionBench.

Vendor-published observation, distinct from independent and other vendor harnesses. Source-specific open-d1 public Decision Index 0.2.1 and decision adaptations; model-name sizes are not benchmark scores. Means refer only to each card’s named task panel.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-07 · Revision: 051bcc464b01b9f92942b364d9586b0ef5912432 · Published observations; result reuse terms not specified

Vision · ImajevBench (dev and calibration, 253 rows) as reported by Liquid AI

Card LiquidAI/d1-3B at 051bcc464b01b9f92942b364d9586b0ef5912432; table Vision.

Published table label: ImajevBench (dev and calibration, 253 rows).

Vendor-published observation, distinct from independent and other vendor harnesses. Source-specific open-d1 public Decision Index 0.2.1 and decision adaptations; model-name sizes are not benchmark scores. Means refer only to each card’s named task panel.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-07 · Revision: 051bcc464b01b9f92942b364d9586b0ef5912432 · Published observations; result reuse terms not specified

Vision · Mean as reported by Liquid AI

Card LiquidAI/d1-3B at 051bcc464b01b9f92942b364d9586b0ef5912432; table Vision.

Published table label: Mean.

Vendor-published observation, distinct from independent and other vendor harnesses. Source-specific open-d1 public Decision Index 0.2.1 and decision adaptations; model-name sizes are not benchmark scores. Means refer only to each card’s named task panel.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-07 · Revision: 051bcc464b01b9f92942b364d9586b0ef5912432 · Published observations; result reuse terms not specified

Vision · MMBench as reported by Liquid AI

Card LiquidAI/d1-3B at 051bcc464b01b9f92942b364d9586b0ef5912432; table Vision.

Published table label: MMBench.

Vendor-published observation, distinct from independent and other vendor harnesses. Source-specific open-d1 public Decision Index 0.2.1 and decision adaptations; model-name sizes are not benchmark scores. Means refer only to each card’s named task panel.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-07 · Revision: 051bcc464b01b9f92942b364d9586b0ef5912432 · Published observations; result reuse terms not specified

Vision · MME as reported by Liquid AI

Card LiquidAI/d1-3B at 051bcc464b01b9f92942b364d9586b0ef5912432; table Vision.

Published table label: MME.

Vendor-published observation, distinct from independent and other vendor harnesses. Source-specific open-d1 public Decision Index 0.2.1 and decision adaptations; model-name sizes are not benchmark scores. Means refer only to each card’s named task panel.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-07 · Revision: 051bcc464b01b9f92942b364d9586b0ef5912432 · Published observations; result reuse terms not specified

Vision · MMStar as reported by Liquid AI

Card LiquidAI/d1-3B at 051bcc464b01b9f92942b364d9586b0ef5912432; table Vision.

Published table label: MMStar.

Vendor-published observation, distinct from independent and other vendor harnesses. Source-specific open-d1 public Decision Index 0.2.1 and decision adaptations; model-name sizes are not benchmark scores. Means refer only to each card’s named task panel.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-07 · Revision: 051bcc464b01b9f92942b364d9586b0ef5912432 · Published observations; result reuse terms not specified

Vision · MMVP as reported by Liquid AI

Card LiquidAI/d1-3B at 051bcc464b01b9f92942b364d9586b0ef5912432; table Vision.

Published table label: MMVP.

Vendor-published observation, distinct from independent and other vendor harnesses. Source-specific open-d1 public Decision Index 0.2.1 and decision adaptations; model-name sizes are not benchmark scores. Means refer only to each card’s named task panel.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-07 · Revision: 051bcc464b01b9f92942b364d9586b0ef5912432 · Published observations; result reuse terms not specified

Vision · POPE as reported by Liquid AI

Card LiquidAI/d1-3B at 051bcc464b01b9f92942b364d9586b0ef5912432; table Vision.

Published table label: POPE.

Vendor-published observation, distinct from independent and other vendor harnesses. Source-specific open-d1 public Decision Index 0.2.1 and decision adaptations; model-name sizes are not benchmark scores. Means refer only to each card’s named task panel.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-07 · Revision: 051bcc464b01b9f92942b364d9586b0ef5912432 · Published observations; result reuse terms not specified

Vision · VisualWebBench as reported by Liquid AI

Card LiquidAI/d1-3B at 051bcc464b01b9f92942b364d9586b0ef5912432; table Vision.

Published table label: VisualWebBench.

Vendor-published observation, distinct from independent and other vendor harnesses. Source-specific open-d1 public Decision Index 0.2.1 and decision adaptations; model-name sizes are not benchmark scores. Means refer only to each card’s named task panel.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-07 · Revision: 051bcc464b01b9f92942b364d9586b0ef5912432 · Published observations; result reuse terms not specified

Vision · VL-RewardBench as reported by Liquid AI

Card LiquidAI/d1-3B at 051bcc464b01b9f92942b364d9586b0ef5912432; table Vision.

Published table label: VL-RewardBench.

Vendor-published observation, distinct from independent and other vendor harnesses. Source-specific open-d1 public Decision Index 0.2.1 and decision adaptations; model-name sizes are not benchmark scores. Means refer only to each card’s named task panel.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-07 · Revision: 051bcc464b01b9f92942b364d9586b0ef5912432 · Published observations; result reuse terms not specified

OneJev peer figures · Public index

OneJev launch figure, peer-reported 16 October board observation.

Vendor-run Decision Index 0.3 public kit; 37 benchmarks / 140,178 requests as reported. Source quotes peers from the 16 October board. Not AA, evaluator-private Full, or S1MB. Exact execution dates, budgets and individual run variances are not published.

Vendor-run Decision Index 0.3 public kit; 37 benchmarks / 140,178 requests as reported. Source quotes peers from the 16 October board. Not AA, evaluator-private Full, or S1MB. Exact execution dates, budgets and individual run variances are not published.

Checked / reported: 2026-10-18 · Samples: Not published

Snapshot: 2026-10-17 · Revision: OneJev 1.6 launch · 2026-10-17 · Published aggregate observations; result reuse terms not separately specified

Accuracy under third-party joint-request scorer, distinct from original hosted d1/submitter-specific ECE conventions.

Exact named LiquidAI/d1-3B; zero-shot HF Jobs, vLLM nightly /v1/systemone, --max-model-len 8192 --max-num-seqs 8; one request per case, 400 cases with all 5 questions each; hardware varies and latency is not ranked. Exact weights SHA and vLLM revision unspecified by source summary.

Third-party SaylorTwift HF Jobs public evaluation, 16 October 2026; accepted into publisher card 19–20 October. Test: 400 cases / 2000 typed decisions; all five questions with the state in one /v1/systemone request. Argmax accuracy counts ties in gold as correct; KL(gold||prediction) clips predictions at 1e-9; native multiclass Brier; ECE over 10 equal-width confidence bins. Same public scorer across these 10 model rows; not rerun by dataset owner. This scorer differs from the older hosted d1 row, so separate dataset IDs. Heterogeneous GPU latency excluded. Teacher-produced soft labels define agreement, not objective correctness. Model checkpoint revision and exact vLLM nightly build are not published in the summary; exact named HF checkpoint, no sibling transfers. Submission: https://huggingface.co/datasets/LocalLLaMA/typed-decisions/discussions/37

Third-party SaylorTwift HF Jobs public evaluation, 16 October 2026; accepted into publisher card 19–20 October. Test: 400 cases / 2000 typed decisions; all five questions with the state in one /v1/systemone request. Argmax accuracy counts ties in gold as correct; KL(gold||prediction) clips predictions at 1e-9; native multiclass Brier; ECE over 10 equal-width confidence bins. Same public scorer across these 10 model rows; not rerun by dataset owner. This scorer differs from the older hosted d1 row, so separate dataset IDs. Heterogeneous GPU latency excluded. Teacher-produced soft labels define agreement, not objective correctness. Model checkpoint revision and exact vLLM nightly build are not published in the summary; exact named HF checkpoint, no sibling transfers.

Checked / reported: 2026-10-20 · Evaluated: 2026-10-16 · Samples: 2000

Snapshot: 2026-10-19T22:54:08.000Z · Revision: d9eda7f31ba227333213a1787f2c78fdaa4090c8 · Apache-2.0 dataset/card; third-party published observations

KL from gold under third-party joint-request scorer, distinct from original hosted d1/submitter-specific ECE conventions.

Exact named LiquidAI/d1-3B; zero-shot HF Jobs, vLLM nightly /v1/systemone, --max-model-len 8192 --max-num-seqs 8; one request per case, 400 cases with all 5 questions each; hardware varies and latency is not ranked. Exact weights SHA and vLLM revision unspecified by source summary.

Third-party SaylorTwift HF Jobs public evaluation, 16 October 2026; accepted into publisher card 19–20 October. Test: 400 cases / 2000 typed decisions; all five questions with the state in one /v1/systemone request. Argmax accuracy counts ties in gold as correct; KL(gold||prediction) clips predictions at 1e-9; native multiclass Brier; ECE over 10 equal-width confidence bins. Same public scorer across these 10 model rows; not rerun by dataset owner. This scorer differs from the older hosted d1 row, so separate dataset IDs. Heterogeneous GPU latency excluded. Teacher-produced soft labels define agreement, not objective correctness. Model checkpoint revision and exact vLLM nightly build are not published in the summary; exact named HF checkpoint, no sibling transfers. Submission: https://huggingface.co/datasets/LocalLLaMA/typed-decisions/discussions/37

Third-party SaylorTwift HF Jobs public evaluation, 16 October 2026; accepted into publisher card 19–20 October. Test: 400 cases / 2000 typed decisions; all five questions with the state in one /v1/systemone request. Argmax accuracy counts ties in gold as correct; KL(gold||prediction) clips predictions at 1e-9; native multiclass Brier; ECE over 10 equal-width confidence bins. Same public scorer across these 10 model rows; not rerun by dataset owner. This scorer differs from the older hosted d1 row, so separate dataset IDs. Heterogeneous GPU latency excluded. Teacher-produced soft labels define agreement, not objective correctness. Model checkpoint revision and exact vLLM nightly build are not published in the summary; exact named HF checkpoint, no sibling transfers.

Checked / reported: 2026-10-20 · Evaluated: 2026-10-16 · Samples: 2000

Snapshot: 2026-10-19T22:54:08.000Z · Revision: d9eda7f31ba227333213a1787f2c78fdaa4090c8 · Apache-2.0 dataset/card; third-party published observations

Brier under third-party joint-request scorer, distinct from original hosted d1/submitter-specific ECE conventions.

Exact named LiquidAI/d1-3B; zero-shot HF Jobs, vLLM nightly /v1/systemone, --max-model-len 8192 --max-num-seqs 8; one request per case, 400 cases with all 5 questions each; hardware varies and latency is not ranked. Exact weights SHA and vLLM revision unspecified by source summary.

Third-party SaylorTwift HF Jobs public evaluation, 16 October 2026; accepted into publisher card 19–20 October. Test: 400 cases / 2000 typed decisions; all five questions with the state in one /v1/systemone request. Argmax accuracy counts ties in gold as correct; KL(gold||prediction) clips predictions at 1e-9; native multiclass Brier; ECE over 10 equal-width confidence bins. Same public scorer across these 10 model rows; not rerun by dataset owner. This scorer differs from the older hosted d1 row, so separate dataset IDs. Heterogeneous GPU latency excluded. Teacher-produced soft labels define agreement, not objective correctness. Model checkpoint revision and exact vLLM nightly build are not published in the summary; exact named HF checkpoint, no sibling transfers. Submission: https://huggingface.co/datasets/LocalLLaMA/typed-decisions/discussions/37

Third-party SaylorTwift HF Jobs public evaluation, 16 October 2026; accepted into publisher card 19–20 October. Test: 400 cases / 2000 typed decisions; all five questions with the state in one /v1/systemone request. Argmax accuracy counts ties in gold as correct; KL(gold||prediction) clips predictions at 1e-9; native multiclass Brier; ECE over 10 equal-width confidence bins. Same public scorer across these 10 model rows; not rerun by dataset owner. This scorer differs from the older hosted d1 row, so separate dataset IDs. Heterogeneous GPU latency excluded. Teacher-produced soft labels define agreement, not objective correctness. Model checkpoint revision and exact vLLM nightly build are not published in the summary; exact named HF checkpoint, no sibling transfers.

Checked / reported: 2026-10-20 · Evaluated: 2026-10-16 · Samples: 2000

Snapshot: 2026-10-19T22:54:08.000Z · Revision: d9eda7f31ba227333213a1787f2c78fdaa4090c8 · Apache-2.0 dataset/card; third-party published observations

ECE under third-party joint-request scorer, distinct from original hosted d1/submitter-specific ECE conventions.

Exact named LiquidAI/d1-3B; zero-shot HF Jobs, vLLM nightly /v1/systemone, --max-model-len 8192 --max-num-seqs 8; one request per case, 400 cases with all 5 questions each; hardware varies and latency is not ranked. Exact weights SHA and vLLM revision unspecified by source summary.

Third-party SaylorTwift HF Jobs public evaluation, 16 October 2026; accepted into publisher card 19–20 October. Test: 400 cases / 2000 typed decisions; all five questions with the state in one /v1/systemone request. Argmax accuracy counts ties in gold as correct; KL(gold||prediction) clips predictions at 1e-9; native multiclass Brier; ECE over 10 equal-width confidence bins. Same public scorer across these 10 model rows; not rerun by dataset owner. This scorer differs from the older hosted d1 row, so separate dataset IDs. Heterogeneous GPU latency excluded. Teacher-produced soft labels define agreement, not objective correctness. Model checkpoint revision and exact vLLM nightly build are not published in the summary; exact named HF checkpoint, no sibling transfers. Submission: https://huggingface.co/datasets/LocalLLaMA/typed-decisions/discussions/37

Third-party SaylorTwift HF Jobs public evaluation, 16 October 2026; accepted into publisher card 19–20 October. Test: 400 cases / 2000 typed decisions; all five questions with the state in one /v1/systemone request. Argmax accuracy counts ties in gold as correct; KL(gold||prediction) clips predictions at 1e-9; native multiclass Brier; ECE over 10 equal-width confidence bins. Same public scorer across these 10 model rows; not rerun by dataset owner. This scorer differs from the older hosted d1 row, so separate dataset IDs. Heterogeneous GPU latency excluded. Teacher-produced soft labels define agreement, not objective correctness. Model checkpoint revision and exact vLLM nightly build are not published in the summary; exact named HF checkpoint, no sibling transfers.

Checked / reported: 2026-10-20 · Evaluated: 2026-10-16 · Samples: 2000

Snapshot: 2026-10-19T22:54:08.000Z · Revision: d9eda7f31ba227333213a1787f2c78fdaa4090c8 · Apache-2.0 dataset/card; third-party published observations