Examenos

Four equally weighted evaluations: AA-Omniscience-Decision, HLE-MultiChoice (513 text-only questions), When2Call and WildGuardTest.

Exact named AA decision system; index v1.0. Served precision and detailed execution date not published in inspected chart.

Rounded chart label manually verified in source; not the ordinary language-model Intelligence Index.

Independent AA results. Omniscience splits its 25% between accuracy and non-hallucination. Displayed integer chart values, not undisclosed precise values. No automated AA import or refresh adapter.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-17 · Revision: Manually checked chart · 2026-10-17 · Published observations; result reuse terms not specified

JevBench Binary slices: aegis2 · acc @0.5

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 82.0%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: aegis2 · acc if flipped

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 18.0%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: aegis2 · AUROC

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.9116.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: aegis2 · Brier

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.1334.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: aegis2 · ECE

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.0621.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: aegis2 · FPR

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 7.6%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: aegis2 · positive recall

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 71.6%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: aegis2_response · acc @0.5

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 82.2%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: aegis2_response · acc if flipped

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 17.8%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: aegis2_response · AUROC

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.9011.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: aegis2_response · Brier

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.1295.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: aegis2_response · ECE

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.0234.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: aegis2_response · FPR

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 17.2%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: aegis2_response · positive recall

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 81.6%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: atbench500 · acc @0.5

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 86.8%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: atbench500 · acc if flipped

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 13.2%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: atbench500 · AUROC

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.9928.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: atbench500 · Brier

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.0877.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: atbench500 · ECE

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.0330.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: atbench500 · FPR

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.4%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: atbench500 · positive recall

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 74.0%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: jailbreak_classification · acc @0.5

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 96.5%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 400

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: jailbreak_classification · acc if flipped

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 3.5%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 400

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: jailbreak_classification · AUROC

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.9939.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 400

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: jailbreak_classification · Brier

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.0317.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 400

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: jailbreak_classification · ECE

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.0321.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 400

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: jailbreak_classification · FPR

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 3.9%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 400

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: jailbreak_classification · positive recall

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 97.2%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 400

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: prompt_injections · acc @0.5

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 73.3%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 116

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: prompt_injections · acc if flipped

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 26.7%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 116

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: prompt_injections · AUROC

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.9875.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 116

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: prompt_injections · Brier

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.1993.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 116

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: prompt_injections · ECE

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.1845.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 116

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: prompt_injections · FPR

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.0%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 116

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: prompt_injections · positive recall

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 48.3%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 116

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench By subset: rewardbench2 · accuracy

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 69.2%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 240

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench By subset: rewardbench2 · accuracy

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 94.2%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 240

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench By subset: rewardbench2 · accuracy

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 73.8%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 240

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench By subset: rewardbench2 · accuracy

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 45.4%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 240

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench By subset: rewardbench2 · accuracy

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 92.1%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 240

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: banking77 · accuracy

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 86.4%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 1000

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: banking77 · Brier

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.1896.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 1000

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: banking77 · ECE (10 bins)

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.0235.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 1000

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: banking77 · macro-F1

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.8590.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 1000

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: banking77 · NLL

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.4587.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 1000

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: mcpbench_server_routing · accuracy

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 94.6%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 112

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: mcpbench_server_routing · Brier

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.0844.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 112

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: mcpbench_server_routing · ECE (10 bins)

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.0215.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 112

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: mcpbench_server_routing · macro-F1

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.9421.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 112

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: mcpbench_server_routing · NLL

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.1893.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 112

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: mcpbench_tool_selection · accuracy

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 100.0%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 26

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: mcpbench_tool_selection · Brier

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.0001.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 26

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: mcpbench_tool_selection · ECE (10 bins)

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.0036.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 26

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: mcpbench_tool_selection · macro-F1

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 1.0000.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 26

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: mcpbench_tool_selection · NLL

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.0036.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 26

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: mcpmark_server_routing · accuracy

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 48.8%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 338

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: mcpmark_server_routing · Brier

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.7481.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 338

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: mcpmark_server_routing · ECE (10 bins)

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.2566.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 338

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: mcpmark_server_routing · macro-F1

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.4185.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 338

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: mcpmark_server_routing · NLL

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 1.8351.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 338

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: mcpmark_tool_selection · accuracy

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 100.0%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 18

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: mcpmark_tool_selection · Brier

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.0000.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 18

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: mcpmark_tool_selection · ECE (10 bins)

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.0015.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 18

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: mcpmark_tool_selection · macro-F1

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 1.0000.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 18

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: mcpmark_tool_selection · NLL

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.0015.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 18

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: medmcqa · accuracy

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 75.9%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 1000

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: medmcqa · Brier

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.3380.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 1000

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: medmcqa · ECE (10 bins)

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.0519.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 1000

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: medmcqa · macro-F1

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.7594.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 1000

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: medmcqa · NLL

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.6393.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 1000

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: medqa_usmle · accuracy

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 80.7%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 1000

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: medqa_usmle · Brier

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.2791.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 1000

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: medqa_usmle · ECE (10 bins)

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.0604.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 1000

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: medqa_usmle · macro-F1

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.8073.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 1000

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: medqa_usmle · NLL

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.5275.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 1000

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: mmlu_pro · accuracy

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 64.7%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 1000

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: mmlu_pro · Brier

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.4933.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 1000

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: mmlu_pro · ECE (10 bins)

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.0643.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 1000

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: mmlu_pro · macro-F1

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.6449.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 1000

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: mmlu_pro · NLL

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 1.1581.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 1000

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: pubmedqa · accuracy

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 68.0%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: pubmedqa · Brier

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.4974.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: pubmedqa · ECE (10 bins)

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.1597.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: pubmedqa · macro-F1

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.6013.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: pubmedqa · NLL

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.9720.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: rewardbench2 · accuracy

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 74.9%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 1200

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: rewardbench2 · Brier

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.3556.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 1200

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: rewardbench2 · ECE (10 bins)

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.0207.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 1200

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: rewardbench2 · macro-F1

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.7494.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 1200

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: rewardbench2 · NLL

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.6922.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 1200

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: scienceqa_text · accuracy

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 93.2%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 1000

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: scienceqa_text · Brier

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.0896.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 1000

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: scienceqa_text · ECE (10 bins)

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.0137.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 1000

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: scienceqa_text · macro-F1

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.9490.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 1000

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: scienceqa_text · NLL

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.1496.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 1000

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Headline: aegis2 · accuracy

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.8200.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 500 · Source 95% CI: 78.4–85.2

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Headline: aegis2_response · accuracy

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.8220.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 500 · Source 95% CI: 78.8–85.6

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Headline: atbench500 · accuracy

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.8680.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 500 · Source 95% CI: 83.8–89.8

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Headline: banking77 · accuracy

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.8640.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 1000 · Source 95% CI: 83.39999999999999–89.3

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Headline: helpsteer2_helpfulness · mae_expected_level

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.9012.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 500 · Source 95% CI: 0.835–0.974

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Headline: jailbreak_classification · accuracy

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.9650.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 400 · Source 95% CI: 94.8–98

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Headline: mcpbench_server_routing · server_routing_accuracy

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.9464.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 112 · Source 95% CI: 87.5–100

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Headline: mcpbench_tool_selection · tool_selection_accuracy

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 1.0000.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 26

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Headline: mcpmark_server_routing · server_routing_accuracy

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.4882.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 338 · Source 95% CI: 41.699999999999996–56.49999999999999

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Headline: mcpmark_tool_selection · tool_selection_accuracy

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 1.0000.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 18

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Headline: medmcqa · accuracy

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.7590.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 1000 · Source 95% CI: 72.2–79.2

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Headline: medqa_usmle · accuracy

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.8070.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 1000 · Source 95% CI: 77.60000000000001–83.89999999999999

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Headline: mmlu_pro · accuracy

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.6470.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 1000 · Source 95% CI: 60.9–68.60000000000001

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Headline: nfcorpus_rank_choice · ndcg@10

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.3644.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 298 · Source 95% CI: 33–39.900000000000006

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Headline: nfcorpus_rank_graded · ndcg@10

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.3732.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 4924 · Source 95% CI: 33.900000000000006–41.099999999999994

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Headline: nfcorpus_rank_noul · ndcg@10

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.3680.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 298 · Source 95% CI: 33.2–40.400000000000006

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Headline: prompt_injections · accuracy

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.7328.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 116 · Source 95% CI: 64.7–81.89999999999999

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Headline: pubmedqa · accuracy

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.6800.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 500 · Source 95% CI: 64–72

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Headline: rewardbench2 · accuracy

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.7492.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 1200 · Source 95% CI: 71.8–77.9

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Headline: scienceqa_text · accuracy

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.9320.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 1000 · Source 95% CI: 91.10000000000001–95.1

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Headline: scifact_rank_choice · ndcg@10

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.7370.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 300 · Source 95% CI: 69.3–78

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Headline: scifact_rank_noul · ndcg@10

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.7302.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 300 · Source 95% CI: 68.4–77.3

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Headline: sst5 · mae_expected_level

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.5418.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 500 · Source 95% CI: 0.505–0.583

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: banking77 · both correct

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 85.8%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: banking77 · flipped by reordering

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 1.2%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: banking77 · original order

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 86.6%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: banking77 · permuted order

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 86.2%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: mcpbench_server_routing · both correct

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 94.6%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 56

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: mcpbench_server_routing · flipped by reordering

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.0%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 56

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: mcpbench_server_routing · original order

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 94.6%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 56

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: mcpbench_server_routing · permuted order

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 94.6%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 56

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: mcpbench_tool_selection · both correct

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 100.0%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 4

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: mcpbench_tool_selection · flipped by reordering

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.0%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 4

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: mcpbench_tool_selection · original order

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 100.0%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 4

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: mcpbench_tool_selection · permuted order

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 100.0%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 4

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: mcpmark_server_routing · both correct

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 47.9%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 169

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: mcpmark_server_routing · flipped by reordering

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 1.8%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 169

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: mcpmark_server_routing · original order

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 49.1%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 169

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: mcpmark_server_routing · permuted order

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 48.5%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 169

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: mcpmark_tool_selection · both correct

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 100.0%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 3

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: mcpmark_tool_selection · flipped by reordering

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.0%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 3

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: mcpmark_tool_selection · original order

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 100.0%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 3

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: mcpmark_tool_selection · permuted order

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 100.0%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 3

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: medmcqa · both correct

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 71.0%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: medmcqa · flipped by reordering

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 9.8%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: medmcqa · original order

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 75.6%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: medmcqa · permuted order

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 76.2%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: medqa_usmle · both correct

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 76.8%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: medqa_usmle · flipped by reordering

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 7.8%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: medqa_usmle · original order

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 80.2%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: medqa_usmle · permuted order

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 81.2%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: mmlu_pro · both correct

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 58.0%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: mmlu_pro · flipped by reordering

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 13.4%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: mmlu_pro · original order

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 64.6%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: mmlu_pro · permuted order

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 64.8%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: rewardbench2 · both correct

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 66.5%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 594

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: rewardbench2 · flipped by reordering

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 16.3%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 594

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: rewardbench2 · original order

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 75.0%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 594

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: rewardbench2 · permuted order

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 74.8%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 594

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: scienceqa_text · both correct

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 90.8%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: scienceqa_text · flipped by reordering

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 4.8%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: scienceqa_text · original order

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 93.2%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: scienceqa_text · permuted order

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 93.2%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ordinal slices: helpsteer2_helpfulness · exact (rounded)

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 35.0%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ordinal slices: helpsteer2_helpfulness · MAE (expected level)

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.901.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ordinal slices: helpsteer2_helpfulness · RPS

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.1591.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ordinal slices: helpsteer2_helpfulness · Spearman

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.566.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ordinal slices: helpsteer2_helpfulness · within 1 level

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 81.4%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ordinal slices: nfcorpus_rank_graded · exact (rounded)

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 71.0%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 4924

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ordinal slices: nfcorpus_rank_graded · MAE (expected level)

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.345.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 4924

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ordinal slices: nfcorpus_rank_graded · RPS

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.1198.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 4924

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ordinal slices: nfcorpus_rank_graded · Spearman

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.490.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 4924

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ordinal slices: nfcorpus_rank_graded · within 1 level

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 97.9%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 4924

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ordinal slices: sst5 · exact (rounded)

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 56.0%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ordinal slices: sst5 · MAE (expected level)

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.542.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ordinal slices: sst5 · RPS

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.0866.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ordinal slices: sst5 · Spearman

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.874.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ordinal slices: sst5 · within 1 level

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 97.4%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ranking slices: nfcorpus_rank_choice · MRR@10

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.6069.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 288

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ranking slices: nfcorpus_rank_choice · nDCG@10

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.3644.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 288

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ranking slices: nfcorpus_rank_choice · Recall@10

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.1669.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 288

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ranking slices: nfcorpus_rank_graded · MRR@10

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.6119.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 298

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ranking slices: nfcorpus_rank_graded · nDCG@10

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.3732.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 298

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ranking slices: nfcorpus_rank_graded · Recall@10

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.1727.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 298

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ranking slices: nfcorpus_rank_noul · MRR@10

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.6080.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 298

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ranking slices: nfcorpus_rank_noul · nDCG@10

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.3680.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 298

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ranking slices: nfcorpus_rank_noul · Recall@10

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.1719.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 298

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ranking slices: scifact_rank_choice · MRR@10

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.7230.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 300

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ranking slices: scifact_rank_choice · nDCG@10

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.7370.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 300

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ranking slices: scifact_rank_choice · Recall@10

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.8101.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 300

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ranking slices: scifact_rank_noul · MRR@10

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.7119.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 300

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ranking slices: scifact_rank_noul · nDCG@10

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.7302.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 300

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ranking slices: scifact_rank_noul · Recall@10

Scored 2026-10-15T23:39:32-0700. Model requested `or-microsoft-microsoft-decision-1`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.8134.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:32-0700 · Samples: 300

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

Liquid vendor’s 112 unique evaluations grouped as Average; not an independent index.

Liquid vendor-run October 15 panel. Exact d1-ultra/Jev 1.14/d3 labels; remaining endpoint revisions and served precision unspecified.

Figure-only printed quantity. Source does not publish per-task weights, complete item populations or all inference settings. Average is the vendor’s published aggregate, never recomputed from rounded domains. Peer model exact hosted revisions can be unspecified.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-15 · Revision: Release figures 2026-10-15 · Published observations; result reuse terms not specified

Liquid vendor’s 112 unique evaluations grouped as Business and new fields; not an independent index.

Liquid vendor-run October 15 panel. Exact d1-ultra/Jev 1.14/d3 labels; remaining endpoint revisions and served precision unspecified.

Figure-only printed quantity. Source does not publish per-task weights, complete item populations or all inference settings. Average is the vendor’s published aggregate, never recomputed from rounded domains. Peer model exact hosted revisions can be unspecified.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-15 · Revision: Release figures 2026-10-15 · Published observations; result reuse terms not specified

Liquid vendor’s 112 unique evaluations grouped as Judgment and taste; not an independent index.

Liquid vendor-run October 15 panel. Exact d1-ultra/Jev 1.14/d3 labels; remaining endpoint revisions and served precision unspecified.

Figure-only printed quantity. Source does not publish per-task weights, complete item populations or all inference settings. Average is the vendor’s published aggregate, never recomputed from rounded domains. Peer model exact hosted revisions can be unspecified.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-15 · Revision: Release figures 2026-10-15 · Published observations; result reuse terms not specified

Liquid vendor’s 112 unique evaluations grouped as Knowledge and reasoning; not an independent index.

Liquid vendor-run October 15 panel. Exact d1-ultra/Jev 1.14/d3 labels; remaining endpoint revisions and served precision unspecified.

Figure-only printed quantity. Source does not publish per-task weights, complete item populations or all inference settings. Average is the vendor’s published aggregate, never recomputed from rounded domains. Peer model exact hosted revisions can be unspecified.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-15 · Revision: Release figures 2026-10-15 · Published observations; result reuse terms not specified

Liquid vendor’s 112 unique evaluations grouped as Languages; not an independent index.

Liquid vendor-run October 15 panel. Exact d1-ultra/Jev 1.14/d3 labels; remaining endpoint revisions and served precision unspecified.

Figure-only printed quantity. Source does not publish per-task weights, complete item populations or all inference settings. Average is the vendor’s published aggregate, never recomputed from rounded domains. Peer model exact hosted revisions can be unspecified.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-15 · Revision: Release figures 2026-10-15 · Published observations; result reuse terms not specified

Liquid vendor’s 112 unique evaluations grouped as Robustness; not an independent index.

Liquid vendor-run October 15 panel. Exact d1-ultra/Jev 1.14/d3 labels; remaining endpoint revisions and served precision unspecified.

Figure-only printed quantity. Source does not publish per-task weights, complete item populations or all inference settings. Average is the vendor’s published aggregate, never recomputed from rounded domains. Peer model exact hosted revisions can be unspecified.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-15 · Revision: Release figures 2026-10-15 · Published observations; result reuse terms not specified

Liquid vendor’s 112 unique evaluations grouped as Safety and policy; not an independent index.

Liquid vendor-run October 15 panel. Exact d1-ultra/Jev 1.14/d3 labels; remaining endpoint revisions and served precision unspecified.

Figure-only printed quantity. Source does not publish per-task weights, complete item populations or all inference settings. Average is the vendor’s published aggregate, never recomputed from rounded domains. Peer model exact hosted revisions can be unspecified.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-15 · Revision: Release figures 2026-10-15 · Published observations; result reuse terms not specified

Liquid vendor’s 112 unique evaluations grouped as Tools and agents; not an independent index.

Liquid vendor-run October 15 panel. Exact d1-ultra/Jev 1.14/d3 labels; remaining endpoint revisions and served precision unspecified.

Figure-only printed quantity. Source does not publish per-task weights, complete item populations or all inference settings. Average is the vendor’s published aggregate, never recomputed from rounded domains. Peer model exact hosted revisions can be unspecified.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-15 · Revision: Release figures 2026-10-15 · Published observations; result reuse terms not specified

Vendor-defined mean of absolute accuracy minus chosen confidence; not interchangeable with other studies’ ECE.

Liquid vendor figure: mean absolute accuracy−confidence on chosen option; hardware/precision unpublished.

Printed figure values manually inspected. Curve coordinates are not reconstructed. Source task weighting and binning are not fully published.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-15 · Revision: Release figure 2026-10-15 · Published observations; result reuse terms not specified

Exact named hosted API observation; All-case accuracy.

Named source route microsoft/microsoft-decision-1; ModelTested runner v1; tested_on=openrouter. Exact backend weight revision and API inference precision unpublished.

Publisher generated 2026-10-18. Execution dates, token usage and hashes remain in its per-run receipts. No private prompt/response archive imported.

56 authored situations, including 19 hard cases, each with two typed questions; 14 public and 42 hidden cases. Source aggregate averages per-case decision accuracy. Confidence score is 100×(1−mean multiclass Brier/2), not ECE or a calibration probability. Small single-run benchmark; current HF access/size does not inherit this evaluator’s stale closed/API-only metadata. No tariff inferred from rounded cost 0.0.

Checked / reported: 2026-10-18 · Samples: 56

Snapshot: 2026-10-18 · Revision: 081e8f37130a4a9077b8cffe62827d4c3b9e3b23 · Published aggregate observations; result reuse terms not separately specified

Exact named hosted API observation; Hard-case accuracy.

Named source route microsoft/microsoft-decision-1; ModelTested runner v1; tested_on=openrouter. Exact backend weight revision and API inference precision unpublished.

Publisher generated 2026-10-18. Execution dates, token usage and hashes remain in its per-run receipts. No private prompt/response archive imported.

56 authored situations, including 19 hard cases, each with two typed questions; 14 public and 42 hidden cases. Source aggregate averages per-case decision accuracy. Confidence score is 100×(1−mean multiclass Brier/2), not ECE or a calibration probability. Small single-run benchmark; current HF access/size does not inherit this evaluator’s stale closed/API-only metadata. No tariff inferred from rounded cost 0.0.

Checked / reported: 2026-10-18 · Samples: 19

Snapshot: 2026-10-18 · Revision: 081e8f37130a4a9077b8cffe62827d4c3b9e3b23 · Published aggregate observations; result reuse terms not separately specified

Exact named hosted API observation; Public-case accuracy.

Named source route microsoft/microsoft-decision-1; ModelTested runner v1; tested_on=openrouter. Exact backend weight revision and API inference precision unpublished.

Publisher generated 2026-10-18. Execution dates, token usage and hashes remain in its per-run receipts. No private prompt/response archive imported.

56 authored situations, including 19 hard cases, each with two typed questions; 14 public and 42 hidden cases. Source aggregate averages per-case decision accuracy. Confidence score is 100×(1−mean multiclass Brier/2), not ECE or a calibration probability. Small single-run benchmark; current HF access/size does not inherit this evaluator’s stale closed/API-only metadata. No tariff inferred from rounded cost 0.0.

Checked / reported: 2026-10-18 · Samples: 14

Snapshot: 2026-10-18 · Revision: 081e8f37130a4a9077b8cffe62827d4c3b9e3b23 · Published aggregate observations; result reuse terms not separately specified

Exact named hosted API observation; Hidden-case accuracy.

Named source route microsoft/microsoft-decision-1; ModelTested runner v1; tested_on=openrouter. Exact backend weight revision and API inference precision unpublished.

Publisher generated 2026-10-18. Execution dates, token usage and hashes remain in its per-run receipts. No private prompt/response archive imported.

56 authored situations, including 19 hard cases, each with two typed questions; 14 public and 42 hidden cases. Source aggregate averages per-case decision accuracy. Confidence score is 100×(1−mean multiclass Brier/2), not ECE or a calibration probability. Small single-run benchmark; current HF access/size does not inherit this evaluator’s stale closed/API-only metadata. No tariff inferred from rounded cost 0.0.

Checked / reported: 2026-10-18 · Samples: 42

Snapshot: 2026-10-18 · Revision: 081e8f37130a4a9077b8cffe62827d4c3b9e3b23 · Published aggregate observations; result reuse terms not separately specified

Exact named hosted API observation; Brier-based confidence score.

Named source route microsoft/microsoft-decision-1; ModelTested runner v1; tested_on=openrouter. Exact backend weight revision and API inference precision unpublished.

Publisher generated 2026-10-18. Execution dates, token usage and hashes remain in its per-run receipts. No private prompt/response archive imported.

56 authored situations, including 19 hard cases, each with two typed questions; 14 public and 42 hidden cases. Source aggregate averages per-case decision accuracy. Confidence score is 100×(1−mean multiclass Brier/2), not ECE or a calibration probability. Small single-run benchmark; current HF access/size does not inherit this evaluator’s stale closed/API-only metadata. No tariff inferred from rounded cost 0.0.

Checked / reported: 2026-10-18 · Samples: 56

Snapshot: 2026-10-18 · Revision: 081e8f37130a4a9077b8cffe62827d4c3b9e3b23 · Published aggregate observations; result reuse terms not separately specified