Mercury Decide Decision Model Results

Mercury Decide

Explore Mercury Decide: exact decision configuration, published specifications, source-linked benchmarks and API tariffs when available.

Published specifications

Provider: Inception
Version: mercury-decide (hosted; provider label 20260930)
System type: Hosted decision model
Access: Hosted
License: Proprietary
Inputs: text, image
Decision types: Choice, Yes / no, Score
Published size: Not published
Active parameters: Not published
Context / input limit: 65,536 tokens (OpenRouter endpoint)
Announced: 2026-09-30

Model card · Website · OpenRouter

Model notes

Hosted diffusion Decision system; not Mercury 3 chat or a downloadable checkpoint. Official HF search found no exact weights or absolute parameter count. OpenRouter labels release 30 September and the provider 20260930; resolved weight SHA remains unknown. Direct /v1/decisions and OpenRouter System One adapters support independent noul/choice/ordinal-score questions over text/JSON/images. Official image limits: base64 PNG/JPEG/WebP/GIF, up to16 images,10MiB each,20MiB total JSON body; these byte limits are not token context. No chat, streaming or tool calling; generic endpoint tool-choice metadata is not proof of Decisions tool support. Claimed up to14 decisions/s is not a controlled independent latency result. API named endpoints may evolve; each benchmark preserves source date/adapter. Vals observes repeated-input billing as question count grows. Benchmark Heaven cost uses an explicitly dated proxy, not the current paid/free offer. A separate current free offer exists at https://openrouter.ai/inception/mercury-decide:free; rate limits and duration are not established.

Checkpoint revision: Hosted revision not published

← All decision models · Compare this decision system →

Published API tariff

$0.02 per 1M input tokens

Inception Decisions API · checked 2026-10-20

Current official50% launch tariff: $0.02 per million input tokens, regularly$0.04; output tokens free. Source publishes zero cached-input price. Multi-question requests may repeat billed state input, as independently observed by Vals; do not assume one shared-state charge. Expiry unpublished. Separate verified OpenRouter :free offer has zero input/output/cache rates; this is an offer of the same hosted model, not another checkpoint. Vals historical $0.021 tariff and workload costs are dated study observations, not current prices.

No output-token charge does not mean free requests. These are provider tariffs, separate from benchmark workload costs.

608 recorded observations. Published results retain their source and setup; no Examenos evaluation was run. Missing results are not zero. How to read these comparisons →

Independent · Artificial Analysis

Mercury Decide · Independent · Artificial Analysis
Evaluation	Result	Evidence
AA Decision Index v1.0	53	Setup & source Four equally weighted evaluations: AA-Omniscience-Decision, HLE-MultiChoice (513 text-only questions), When2Call and WildGuardTest. Named Mercury Decide hosted Decision endpoint; AA Decisions v1.0. Exact execution dates, precision and serving weight SHA are unpublished. Manual browser observation; no automated AA adapter. Calibration curves have no single published scalar to infer. Independent AA results. Omniscience splits its 25% between accuracy and non-hallucination. Displayed integer chart values, not undisclosed precise values. No automated AA import or refresh adapter. Checked / reported: 2026-10-20 · Samples: Not published Snapshot: 2026-10-17 · Revision: Manually checked chart · 2026-10-17 · Published observations; result reuse terms not specified Result source · Benchmark source
AA Decisions v1 · AA-Omniscience-Decision accuracy	35%	Setup & source AA Decisions v1 · AA-Omniscience-Decision accuracy Named Mercury Decide hosted Decision endpoint; AA Decisions v1.0. Exact execution dates, precision and serving weight SHA are unpublished. Manual browser observation; no automated AA adapter. Calibration curves have no single published scalar to infer. Source-rounded chart label manually verified, not hidden precise data. Five capability components feed the four-benchmark index. Task cost is average workload cost at source current prices, not a token tariff. Time is median across tasks, not a controlled provider benchmark. Checked / reported: 2026-10-20 · Samples: Not published Snapshot: 2026-10-20 · Revision: Manual rendered chart labels · 2026-10-20 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
AA Decisions v1 · AA-Omniscience-Decision non-hallucination	32%	Setup & source AA Decisions v1 · AA-Omniscience-Decision non-hallucination Named Mercury Decide hosted Decision endpoint; AA Decisions v1.0. Exact execution dates, precision and serving weight SHA are unpublished. Manual browser observation; no automated AA adapter. Calibration curves have no single published scalar to infer. Source-rounded chart label manually verified, not hidden precise data. Five capability components feed the four-benchmark index. Task cost is average workload cost at source current prices, not a token tariff. Time is median across tasks, not a controlled provider benchmark. Checked / reported: 2026-10-20 · Samples: Not published Snapshot: 2026-10-20 · Revision: Manual rendered chart labels · 2026-10-20 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
AA Decisions v1 · HLE-MultiChoice	10%	Setup & source AA Decisions v1 · HLE-MultiChoice Named Mercury Decide hosted Decision endpoint; AA Decisions v1.0. Exact execution dates, precision and serving weight SHA are unpublished. Manual browser observation; no automated AA adapter. Calibration curves have no single published scalar to infer. Source-rounded chart label manually verified, not hidden precise data. Five capability components feed the four-benchmark index. Task cost is average workload cost at source current prices, not a token tariff. Time is median across tasks, not a controlled provider benchmark. Checked / reported: 2026-10-20 · Samples: Not published Snapshot: 2026-10-20 · Revision: Manual rendered chart labels · 2026-10-20 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
AA Decisions v1 · When2Call	79%	Setup & source AA Decisions v1 · When2Call Named Mercury Decide hosted Decision endpoint; AA Decisions v1.0. Exact execution dates, precision and serving weight SHA are unpublished. Manual browser observation; no automated AA adapter. Calibration curves have no single published scalar to infer. Source-rounded chart label manually verified, not hidden precise data. Five capability components feed the four-benchmark index. Task cost is average workload cost at source current prices, not a token tariff. Time is median across tasks, not a controlled provider benchmark. Checked / reported: 2026-10-20 · Samples: Not published Snapshot: 2026-10-20 · Revision: Manual rendered chart labels · 2026-10-20 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
AA Decisions v1 · WildGuardTest average F1	88%	Setup & source AA Decisions v1 · WildGuardTest average F1 Named Mercury Decide hosted Decision endpoint; AA Decisions v1.0. Exact execution dates, precision and serving weight SHA are unpublished. Manual browser observation; no automated AA adapter. Calibration curves have no single published scalar to infer. Source-rounded chart label manually verified, not hidden precise data. Five capability components feed the four-benchmark index. Task cost is average workload cost at source current prices, not a token tariff. Time is median across tasks, not a controlled provider benchmark. Checked / reported: 2026-10-20 · Samples: Not published Snapshot: 2026-10-20 · Revision: Manual rendered chart labels · 2026-10-20 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
AA Decisions v1 · Median time per task	500 ms	Setup & source AA Decisions v1 · Median time per task Named Mercury Decide hosted Decision endpoint; AA Decisions v1.0. Exact execution dates, precision and serving weight SHA are unpublished. Manual browser observation; no automated AA adapter. Calibration curves have no single published scalar to infer. Source-rounded chart label manually verified, not hidden precise data. Five capability components feed the four-benchmark index. Task cost is average workload cost at source current prices, not a token tariff. Time is median across tasks, not a controlled provider benchmark. Checked / reported: 2026-10-20 · Samples: Not published Snapshot: 2026-10-20 · Revision: Manual rendered chart labels · 2026-10-20 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
AA Decisions v1 · Cost per million tasks	15 USD/1M tasks	Setup & source AA Decisions v1 · Cost per million tasks Named Mercury Decide hosted Decision endpoint; AA Decisions v1.0. Exact execution dates, precision and serving weight SHA are unpublished. Manual browser observation; no automated AA adapter. Calibration curves have no single published scalar to infer. Source-rounded chart label manually verified, not hidden precise data. Five capability components feed the four-benchmark index. Task cost is average workload cost at source current prices, not a token tariff. Time is median across tasks, not a controlled provider benchmark. Checked / reported: 2026-10-20 · Samples: Not published Snapshot: 2026-10-20 · Revision: Manual rendered chart labels · 2026-10-20 · Published summary observations; upstream task/result terms retained Result source · Benchmark source

Independent · JevBench

Mercury Decide · Independent · JevBench
Evaluation	Result	Evidence
JevBench · aegis2 · accuracy	84.00%	Setup & source JevBench Headline: aegis2 · accuracy Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "aegis2", "type": "noul", "cases": "500", "coverage": "100.0%", "value": "0.8400", "95% CI": "[0.808, 0.870]", "metric": "accuracy"}. Runtime diagnostics {"slice": "aegis2", "p50 ms": "623", "p95 ms": "5289", "p99 ms": "6059", "mean input tokens": "180", "total input tokens": "90,031", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 500 · Source 95% CI: 80.80000000000001–87 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · aegis2_response · accuracy	77.20%	Setup & source JevBench Headline: aegis2_response · accuracy Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "aegis2_response", "type": "noul", "cases": "500", "coverage": "100.0%", "value": "0.7720", "95% CI": "[0.734, 0.810]", "metric": "accuracy"}. Runtime diagnostics {"slice": "aegis2_response", "p50 ms": "564", "p95 ms": "1376", "p99 ms": "1831", "mean input tokens": "282", "total input tokens": "140,800", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 500 · Source 95% CI: 73.4–81 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · atbench500 · accuracy	68.80%	Setup & source JevBench Headline: atbench500 · accuracy Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "atbench500", "type": "noul", "cases": "500", "coverage": "100.0%", "value": "0.6880", "95% CI": "[0.646, 0.730]", "metric": "accuracy"}. Runtime diagnostics {"slice": "atbench500", "p50 ms": "851", "p95 ms": "1790", "p99 ms": "2262", "mean input tokens": "1895", "total input tokens": "947,364", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 500 · Source 95% CI: 64.60000000000001–73 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · banking77 · accuracy	89.90%	Setup & source JevBench Headline: banking77 · accuracy Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "banking77", "type": "choice", "cases": "1000", "coverage": "100.0%", "value": "0.8990", "95% CI": "[0.875, 0.924]", "metric": "accuracy"}. Runtime diagnostics {"slice": "banking77", "p50 ms": "854", "p95 ms": "1951", "p99 ms": "2792", "mean input tokens": "1599", "total input tokens": "1,599,482", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 1000 · Source 95% CI: 87.5–92.4 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · helpsteer2_helpfulness · mae_expected_level	0.9469 MAE	Setup & source JevBench Headline: helpsteer2_helpfulness · mae_expected_level Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "helpsteer2_helpfulness", "type": "score", "cases": "500", "coverage": "100.0%", "value": "0.9469", "95% CI": "[0.878, 1.017]", "metric": "mae_expected_level"}. Runtime diagnostics {"slice": "helpsteer2_helpfulness", "p50 ms": "924", "p95 ms": "6076", "p99 ms": "6843", "mean input tokens": "670", "total input tokens": "335,063", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 500 · Source 95% CI: 0.878–1.017 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · jailbreak_classification · accuracy	96.75%	Setup & source JevBench Headline: jailbreak_classification · accuracy Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "jailbreak_classification", "type": "noul", "cases": "400", "coverage": "100.0%", "value": "0.9675", "95% CI": "[0.950, 0.983]", "metric": "accuracy"}. Runtime diagnostics {"slice": "jailbreak_classification", "p50 ms": "577", "p95 ms": "1288", "p99 ms": "1876", "mean input tokens": "388", "total input tokens": "155,110", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 400 · Source 95% CI: 95–98.3 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · mcpbench_server_routing · server_routing_accuracy	100.00%	Setup & source JevBench Headline: mcpbench_server_routing · server_routing_accuracy Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "mcpbench_server_routing", "type": "choice", "cases": "112", "coverage": "100.0%", "value": "1.0000", "95% CI": "[1.000, 1.000]", "metric": "server_routing_accuracy"}. Runtime diagnostics {"slice": "mcpbench_server_routing", "p50 ms": "852", "p95 ms": "1950", "p99 ms": "2237", "mean input tokens": "867", "total input tokens": "97,113", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 112 · Source 95% CI: 100–100 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · mcpbench_tool_selection · tool_selection_accuracy	100.00%	Setup & source JevBench Headline: mcpbench_tool_selection · tool_selection_accuracy Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "mcpbench_tool_selection", "type": "choice", "cases": "26", "coverage": "100.0%", "value": "1.0000", "95% CI": "n/a", "metric": "tool_selection_accuracy"}. Runtime diagnostics {"slice": "mcpbench_tool_selection", "p50 ms": "773", "p95 ms": "2045", "p99 ms": "2180", "mean input tokens": "755", "total input tokens": "19,632", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 26 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · mcpmark_server_routing · server_routing_accuracy	57.10%	Setup & source JevBench Headline: mcpmark_server_routing · server_routing_accuracy Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "mcpmark_server_routing", "type": "choice", "cases": "338", "coverage": "100.0%", "value": "0.5710", "95% CI": "[0.500, 0.645]", "metric": "server_routing_accuracy"}. Runtime diagnostics {"slice": "mcpmark_server_routing", "p50 ms": "545", "p95 ms": "1223", "p99 ms": "1453", "mean input tokens": "248", "total input tokens": "83,834", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 338 · Source 95% CI: 50–64.5 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · mcpmark_tool_selection · tool_selection_accuracy	100.00%	Setup & source JevBench Headline: mcpmark_tool_selection · tool_selection_accuracy Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "mcpmark_tool_selection", "type": "choice", "cases": "18", "coverage": "100.0%", "value": "1.0000", "95% CI": "n/a", "metric": "tool_selection_accuracy"}. Runtime diagnostics {"slice": "mcpmark_tool_selection", "p50 ms": "683", "p95 ms": "3488", "p99 ms": "3488", "mean input tokens": "402", "total input tokens": "7,245", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 18 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · medmcqa · accuracy	72.60%	Setup & source JevBench Headline: medmcqa · accuracy Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "medmcqa", "type": "choice", "cases": "1000", "coverage": "100.0%", "value": "0.7260", "95% CI": "[0.691, 0.763]", "metric": "accuracy"}. Runtime diagnostics {"slice": "medmcqa", "p50 ms": "590", "p95 ms": "1329", "p99 ms": "1806", "mean input tokens": "141", "total input tokens": "141,057", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 1000 · Source 95% CI: 69.1–76.3 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · medqa_usmle · accuracy	84.90%	Setup & source JevBench Headline: medqa_usmle · accuracy Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "medqa_usmle", "type": "choice", "cases": "1000", "coverage": "100.0%", "value": "0.8490", "95% CI": "[0.820, 0.877]", "metric": "accuracy"}. Runtime diagnostics {"slice": "medqa_usmle", "p50 ms": "596", "p95 ms": "1546", "p99 ms": "2754", "mean input tokens": "301", "total input tokens": "301,460", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 1000 · Source 95% CI: 82–87.7 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · mmlu_pro · accuracy	64.50%	Setup & source JevBench Headline: mmlu_pro · accuracy Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "mmlu_pro", "type": "choice", "cases": "1000", "coverage": "100.0%", "value": "0.6450", "95% CI": "[0.606, 0.683]", "metric": "accuracy"}. Runtime diagnostics {"slice": "mmlu_pro", "p50 ms": "625", "p95 ms": "1712", "p99 ms": "2715", "mean input tokens": "333", "total input tokens": "332,557", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 1000 · Source 95% CI: 60.6–68.30000000000001 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · nfcorpus_rank_choice · ndcg@10	35.54%	Setup & source JevBench Headline: nfcorpus_rank_choice · ndcg@10 Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "nfcorpus_rank_choice", "type": "rank", "cases": "298", "coverage": "100.0%", "value": "0.3554", "95% CI": "[0.321, 0.389]", "metric": "ndcg@10"}. Runtime diagnostics {"slice": "nfcorpus_rank_choice", "p50 ms": "1131", "p95 ms": "6996", "p99 ms": "7522", "mean input tokens": "2380", "total input tokens": "709,228", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 298 · Source 95% CI: 32.1–38.9 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · nfcorpus_rank_graded · ndcg@10	36.73%	Setup & source JevBench Headline: nfcorpus_rank_graded · ndcg@10 Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "nfcorpus_rank_graded", "type": "score", "cases": "4924", "coverage": "100.0%", "value": "0.3673", "95% CI": "[0.333, 0.404]", "metric": "ndcg@10"}. Runtime diagnostics {"slice": "nfcorpus_rank_graded", "p50 ms": "582", "p95 ms": "1449", "p99 ms": "2023", "mean input tokens": "307", "total input tokens": "1,511,895", "retries": "1"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 4924 · Source 95% CI: 33.300000000000004–40.400000000000006 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · nfcorpus_rank_noul · ndcg@10	37.46%	Setup & source JevBench Headline: nfcorpus_rank_noul · ndcg@10 Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "nfcorpus_rank_noul", "type": "rank", "cases": "298", "coverage": "100.0%", "value": "0.3746", "95% CI": "[0.340, 0.412]", "metric": "ndcg@10"}. Runtime diagnostics {"slice": "nfcorpus_rank_noul", "p50 ms": "2641", "p95 ms": "4668", "p99 ms": "5745", "mean input tokens": "2399", "total input tokens": "715,032", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 298 · Source 95% CI: 34–41.199999999999996 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · prompt_injections · accuracy	81.03%	Setup & source JevBench Headline: prompt_injections · accuracy Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "prompt_injections", "type": "noul", "cases": "116", "coverage": "100.0%", "value": "0.8103", "95% CI": "[0.741, 0.879]", "metric": "accuracy"}. Runtime diagnostics {"slice": "prompt_injections", "p50 ms": "996", "p95 ms": "2094", "p99 ms": "3122", "mean input tokens": "171", "total input tokens": "19,796", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 116 · Source 95% CI: 74.1–87.9 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · pubmedqa · accuracy	65.60%	Setup & source JevBench Headline: pubmedqa · accuracy Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "pubmedqa", "type": "choice", "cases": "500", "coverage": "100.0%", "value": "0.6560", "95% CI": "[0.616, 0.698]", "metric": "accuracy"}. Runtime diagnostics {"slice": "pubmedqa", "p50 ms": "1063", "p95 ms": "2672", "p99 ms": "3697", "mean input tokens": "457", "total input tokens": "228,509", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 500 · Source 95% CI: 61.6–69.8 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · rewardbench2 · accuracy	79.33%	Setup & source JevBench Headline: rewardbench2 · accuracy Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "rewardbench2", "type": "choice", "cases": "1200", "coverage": "100.0%", "value": "0.7933", "95% CI": "[0.763, 0.822]", "metric": "accuracy"}. Runtime diagnostics {"slice": "rewardbench2", "p50 ms": "896", "p95 ms": "2057", "p99 ms": "2813", "mean input tokens": "1777", "total input tokens": "2,132,815", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 1200 · Source 95% CI: 76.3–82.19999999999999 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · scienceqa_text · accuracy	94.20%	Setup & source JevBench Headline: scienceqa_text · accuracy Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "scienceqa_text", "type": "choice", "cases": "1000", "coverage": "100.0%", "value": "0.9420", "95% CI": "[0.922, 0.960]", "metric": "accuracy"}. Runtime diagnostics {"slice": "scienceqa_text", "p50 ms": "896", "p95 ms": "2274", "p99 ms": "2831", "mean input tokens": "129", "total input tokens": "129,244", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 1000 · Source 95% CI: 92.2–96 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · scifact_rank_choice · ndcg@10	72.55%	Setup & source JevBench Headline: scifact_rank_choice · ndcg@10 Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "scifact_rank_choice", "type": "rank", "cases": "300", "coverage": "100.0%", "value": "0.7255", "95% CI": "[0.682, 0.769]", "metric": "ndcg@10"}. Runtime diagnostics {"slice": "scifact_rank_choice", "p50 ms": "1042", "p95 ms": "2422", "p99 ms": "2853", "mean input tokens": "3003", "total input tokens": "901,036", "retries": "3"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 300 · Source 95% CI: 68.2–76.9 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · scifact_rank_noul · ndcg@10	74.33%	Setup & source JevBench Headline: scifact_rank_noul · ndcg@10 Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "scifact_rank_noul", "type": "rank", "cases": "300", "coverage": "100.0%", "value": "0.7433", "95% CI": "[0.699, 0.787]", "metric": "ndcg@10"}. Runtime diagnostics {"slice": "scifact_rank_noul", "p50 ms": "3568", "p95 ms": "7529", "p99 ms": "8694", "mean input tokens": "3020", "total input tokens": "905,961", "retries": "312"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 300 · Source 95% CI: 69.89999999999999–78.7 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · sst5 · mae_expected_level	0.4768 MAE	Setup & source JevBench Headline: sst5 · mae_expected_level Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "sst5", "type": "score", "cases": "500", "coverage": "100.0%", "value": "0.4768", "95% CI": "[0.439, 0.516]", "metric": "mae_expected_level"}. Runtime diagnostics {"slice": "sst5", "p50 ms": "833", "p95 ms": "1986", "p99 ms": "2713", "mean input tokens": "139", "total input tokens": "69,251", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 500 · Source 95% CI: 0.439–0.516 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · banking77 · accuracy	89.90%	Setup & source JevBench Choice slices: banking77 · accuracy Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "banking77", "n": "1000", "gold classes": "77", "accuracy": "89.9%", "macro-F1": "0.8962", "Brier": "0.1670", "NLL": "0.4946", "zero-prob on gold": "0", "ECE (10 bins)": "0.0637", "n with probs": "1000"}. Runtime diagnostics {"slice": "banking77", "p50 ms": "854", "p95 ms": "1951", "p99 ms": "2792", "mean input tokens": "1599", "total input tokens": "1,599,482", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 1000 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · banking77 · macro-F1	89.62%	Setup & source JevBench Choice slices: banking77 · macro-F1 Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "banking77", "n": "1000", "gold classes": "77", "accuracy": "89.9%", "macro-F1": "0.8962", "Brier": "0.1670", "NLL": "0.4946", "zero-prob on gold": "0", "ECE (10 bins)": "0.0637", "n with probs": "1000"}. Runtime diagnostics {"slice": "banking77", "p50 ms": "854", "p95 ms": "1951", "p99 ms": "2792", "mean input tokens": "1599", "total input tokens": "1,599,482", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 1000 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · banking77 · Brier	0.1670 Brier	Setup & source JevBench Choice slices: banking77 · Brier Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "banking77", "n": "1000", "gold classes": "77", "accuracy": "89.9%", "macro-F1": "0.8962", "Brier": "0.1670", "NLL": "0.4946", "zero-prob on gold": "0", "ECE (10 bins)": "0.0637", "n with probs": "1000"}. Runtime diagnostics {"slice": "banking77", "p50 ms": "854", "p95 ms": "1951", "p99 ms": "2792", "mean input tokens": "1599", "total input tokens": "1,599,482", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 1000 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · banking77 · NLL	0.4946 NLL	Setup & source JevBench Choice slices: banking77 · NLL Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "banking77", "n": "1000", "gold classes": "77", "accuracy": "89.9%", "macro-F1": "0.8962", "Brier": "0.1670", "NLL": "0.4946", "zero-prob on gold": "0", "ECE (10 bins)": "0.0637", "n with probs": "1000"}. Runtime diagnostics {"slice": "banking77", "p50 ms": "854", "p95 ms": "1951", "p99 ms": "2792", "mean input tokens": "1599", "total input tokens": "1,599,482", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 1000 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · banking77 · ECE (10 bins)	6.37 pp	Setup & source JevBench Choice slices: banking77 · ECE (10 bins) Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "banking77", "n": "1000", "gold classes": "77", "accuracy": "89.9%", "macro-F1": "0.8962", "Brier": "0.1670", "NLL": "0.4946", "zero-prob on gold": "0", "ECE (10 bins)": "0.0637", "n with probs": "1000"}. Runtime diagnostics {"slice": "banking77", "p50 ms": "854", "p95 ms": "1951", "p99 ms": "2792", "mean input tokens": "1599", "total input tokens": "1,599,482", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 1000 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · mcpbench_server_routing · accuracy	100.00%	Setup & source JevBench Choice slices: mcpbench_server_routing · accuracy Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "mcpbench_server_routing", "n": "112", "gold classes": "28", "accuracy": "100.0%", "macro-F1": "1.0000", "Brier": "0.0140", "NLL": "0.0342", "zero-prob on gold": "0", "ECE (10 bins)": "0.0236", "n with probs": "112"}. Runtime diagnostics {"slice": "mcpbench_server_routing", "p50 ms": "852", "p95 ms": "1950", "p99 ms": "2237", "mean input tokens": "867", "total input tokens": "97,113", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 112 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · mcpbench_server_routing · macro-F1	100.00%	Setup & source JevBench Choice slices: mcpbench_server_routing · macro-F1 Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "mcpbench_server_routing", "n": "112", "gold classes": "28", "accuracy": "100.0%", "macro-F1": "1.0000", "Brier": "0.0140", "NLL": "0.0342", "zero-prob on gold": "0", "ECE (10 bins)": "0.0236", "n with probs": "112"}. Runtime diagnostics {"slice": "mcpbench_server_routing", "p50 ms": "852", "p95 ms": "1950", "p99 ms": "2237", "mean input tokens": "867", "total input tokens": "97,113", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 112 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · mcpbench_server_routing · Brier	0.0140 Brier	Setup & source JevBench Choice slices: mcpbench_server_routing · Brier Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "mcpbench_server_routing", "n": "112", "gold classes": "28", "accuracy": "100.0%", "macro-F1": "1.0000", "Brier": "0.0140", "NLL": "0.0342", "zero-prob on gold": "0", "ECE (10 bins)": "0.0236", "n with probs": "112"}. Runtime diagnostics {"slice": "mcpbench_server_routing", "p50 ms": "852", "p95 ms": "1950", "p99 ms": "2237", "mean input tokens": "867", "total input tokens": "97,113", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 112 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · mcpbench_server_routing · NLL	0.0342 NLL	Setup & source JevBench Choice slices: mcpbench_server_routing · NLL Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "mcpbench_server_routing", "n": "112", "gold classes": "28", "accuracy": "100.0%", "macro-F1": "1.0000", "Brier": "0.0140", "NLL": "0.0342", "zero-prob on gold": "0", "ECE (10 bins)": "0.0236", "n with probs": "112"}. Runtime diagnostics {"slice": "mcpbench_server_routing", "p50 ms": "852", "p95 ms": "1950", "p99 ms": "2237", "mean input tokens": "867", "total input tokens": "97,113", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 112 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · mcpbench_server_routing · ECE (10 bins)	2.36 pp	Setup & source JevBench Choice slices: mcpbench_server_routing · ECE (10 bins) Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "mcpbench_server_routing", "n": "112", "gold classes": "28", "accuracy": "100.0%", "macro-F1": "1.0000", "Brier": "0.0140", "NLL": "0.0342", "zero-prob on gold": "0", "ECE (10 bins)": "0.0236", "n with probs": "112"}. Runtime diagnostics {"slice": "mcpbench_server_routing", "p50 ms": "852", "p95 ms": "1950", "p99 ms": "2237", "mean input tokens": "867", "total input tokens": "97,113", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 112 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · mcpbench_tool_selection · accuracy	100.00%	Setup & source JevBench Choice slices: mcpbench_tool_selection · accuracy Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "mcpbench_tool_selection", "n": "26", "gold classes": "13", "accuracy": "100.0%", "macro-F1": "1.0000", "Brier": "0.0000", "NLL": "0.0002", "zero-prob on gold": "0", "ECE (10 bins)": "0.0002", "n with probs": "26"}. Runtime diagnostics {"slice": "mcpbench_tool_selection", "p50 ms": "773", "p95 ms": "2045", "p99 ms": "2180", "mean input tokens": "755", "total input tokens": "19,632", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 26 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · mcpbench_tool_selection · macro-F1	100.00%	Setup & source JevBench Choice slices: mcpbench_tool_selection · macro-F1 Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "mcpbench_tool_selection", "n": "26", "gold classes": "13", "accuracy": "100.0%", "macro-F1": "1.0000", "Brier": "0.0000", "NLL": "0.0002", "zero-prob on gold": "0", "ECE (10 bins)": "0.0002", "n with probs": "26"}. Runtime diagnostics {"slice": "mcpbench_tool_selection", "p50 ms": "773", "p95 ms": "2045", "p99 ms": "2180", "mean input tokens": "755", "total input tokens": "19,632", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 26 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · mcpbench_tool_selection · Brier	0.0000 Brier	Setup & source JevBench Choice slices: mcpbench_tool_selection · Brier Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "mcpbench_tool_selection", "n": "26", "gold classes": "13", "accuracy": "100.0%", "macro-F1": "1.0000", "Brier": "0.0000", "NLL": "0.0002", "zero-prob on gold": "0", "ECE (10 bins)": "0.0002", "n with probs": "26"}. Runtime diagnostics {"slice": "mcpbench_tool_selection", "p50 ms": "773", "p95 ms": "2045", "p99 ms": "2180", "mean input tokens": "755", "total input tokens": "19,632", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 26 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · mcpbench_tool_selection · NLL	0.0002 NLL	Setup & source JevBench Choice slices: mcpbench_tool_selection · NLL Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "mcpbench_tool_selection", "n": "26", "gold classes": "13", "accuracy": "100.0%", "macro-F1": "1.0000", "Brier": "0.0000", "NLL": "0.0002", "zero-prob on gold": "0", "ECE (10 bins)": "0.0002", "n with probs": "26"}. Runtime diagnostics {"slice": "mcpbench_tool_selection", "p50 ms": "773", "p95 ms": "2045", "p99 ms": "2180", "mean input tokens": "755", "total input tokens": "19,632", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 26 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · mcpbench_tool_selection · ECE (10 bins)	0.02 pp	Setup & source JevBench Choice slices: mcpbench_tool_selection · ECE (10 bins) Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "mcpbench_tool_selection", "n": "26", "gold classes": "13", "accuracy": "100.0%", "macro-F1": "1.0000", "Brier": "0.0000", "NLL": "0.0002", "zero-prob on gold": "0", "ECE (10 bins)": "0.0002", "n with probs": "26"}. Runtime diagnostics {"slice": "mcpbench_tool_selection", "p50 ms": "773", "p95 ms": "2045", "p99 ms": "2180", "mean input tokens": "755", "total input tokens": "19,632", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 26 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · mcpmark_server_routing · accuracy	57.10%	Setup & source JevBench Choice slices: mcpmark_server_routing · accuracy Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "mcpmark_server_routing", "n": "338", "gold classes": "8", "accuracy": "57.1%", "macro-F1": "0.5157", "Brier": "0.6637", "NLL": "1.7849", "zero-prob on gold": "0", "ECE (10 bins)": "0.2803", "n with probs": "338"}. Runtime diagnostics {"slice": "mcpmark_server_routing", "p50 ms": "545", "p95 ms": "1223", "p99 ms": "1453", "mean input tokens": "248", "total input tokens": "83,834", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 338 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · mcpmark_server_routing · macro-F1	51.57%	Setup & source JevBench Choice slices: mcpmark_server_routing · macro-F1 Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "mcpmark_server_routing", "n": "338", "gold classes": "8", "accuracy": "57.1%", "macro-F1": "0.5157", "Brier": "0.6637", "NLL": "1.7849", "zero-prob on gold": "0", "ECE (10 bins)": "0.2803", "n with probs": "338"}. Runtime diagnostics {"slice": "mcpmark_server_routing", "p50 ms": "545", "p95 ms": "1223", "p99 ms": "1453", "mean input tokens": "248", "total input tokens": "83,834", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 338 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · mcpmark_server_routing · Brier	0.6637 Brier	Setup & source JevBench Choice slices: mcpmark_server_routing · Brier Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "mcpmark_server_routing", "n": "338", "gold classes": "8", "accuracy": "57.1%", "macro-F1": "0.5157", "Brier": "0.6637", "NLL": "1.7849", "zero-prob on gold": "0", "ECE (10 bins)": "0.2803", "n with probs": "338"}. Runtime diagnostics {"slice": "mcpmark_server_routing", "p50 ms": "545", "p95 ms": "1223", "p99 ms": "1453", "mean input tokens": "248", "total input tokens": "83,834", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 338 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · mcpmark_server_routing · NLL	1.7849 NLL	Setup & source JevBench Choice slices: mcpmark_server_routing · NLL Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "mcpmark_server_routing", "n": "338", "gold classes": "8", "accuracy": "57.1%", "macro-F1": "0.5157", "Brier": "0.6637", "NLL": "1.7849", "zero-prob on gold": "0", "ECE (10 bins)": "0.2803", "n with probs": "338"}. Runtime diagnostics {"slice": "mcpmark_server_routing", "p50 ms": "545", "p95 ms": "1223", "p99 ms": "1453", "mean input tokens": "248", "total input tokens": "83,834", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 338 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · mcpmark_server_routing · ECE (10 bins)	28.03 pp	Setup & source JevBench Choice slices: mcpmark_server_routing · ECE (10 bins) Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "mcpmark_server_routing", "n": "338", "gold classes": "8", "accuracy": "57.1%", "macro-F1": "0.5157", "Brier": "0.6637", "NLL": "1.7849", "zero-prob on gold": "0", "ECE (10 bins)": "0.2803", "n with probs": "338"}. Runtime diagnostics {"slice": "mcpmark_server_routing", "p50 ms": "545", "p95 ms": "1223", "p99 ms": "1453", "mean input tokens": "248", "total input tokens": "83,834", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 338 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · mcpmark_tool_selection · accuracy	100.00%	Setup & source JevBench Choice slices: mcpmark_tool_selection · accuracy Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "mcpmark_tool_selection", "n": "18", "gold classes": "5", "accuracy": "100.0%", "macro-F1": "1.0000", "Brier": "0.0000", "NLL": "0.0001", "zero-prob on gold": "0", "ECE (10 bins)": "0.0001", "n with probs": "18"}. Runtime diagnostics {"slice": "mcpmark_tool_selection", "p50 ms": "683", "p95 ms": "3488", "p99 ms": "3488", "mean input tokens": "402", "total input tokens": "7,245", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 18 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · mcpmark_tool_selection · macro-F1	100.00%	Setup & source JevBench Choice slices: mcpmark_tool_selection · macro-F1 Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "mcpmark_tool_selection", "n": "18", "gold classes": "5", "accuracy": "100.0%", "macro-F1": "1.0000", "Brier": "0.0000", "NLL": "0.0001", "zero-prob on gold": "0", "ECE (10 bins)": "0.0001", "n with probs": "18"}. Runtime diagnostics {"slice": "mcpmark_tool_selection", "p50 ms": "683", "p95 ms": "3488", "p99 ms": "3488", "mean input tokens": "402", "total input tokens": "7,245", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 18 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · mcpmark_tool_selection · Brier	0.0000 Brier	Setup & source JevBench Choice slices: mcpmark_tool_selection · Brier Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "mcpmark_tool_selection", "n": "18", "gold classes": "5", "accuracy": "100.0%", "macro-F1": "1.0000", "Brier": "0.0000", "NLL": "0.0001", "zero-prob on gold": "0", "ECE (10 bins)": "0.0001", "n with probs": "18"}. Runtime diagnostics {"slice": "mcpmark_tool_selection", "p50 ms": "683", "p95 ms": "3488", "p99 ms": "3488", "mean input tokens": "402", "total input tokens": "7,245", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 18 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · mcpmark_tool_selection · NLL	0.0001 NLL	Setup & source JevBench Choice slices: mcpmark_tool_selection · NLL Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "mcpmark_tool_selection", "n": "18", "gold classes": "5", "accuracy": "100.0%", "macro-F1": "1.0000", "Brier": "0.0000", "NLL": "0.0001", "zero-prob on gold": "0", "ECE (10 bins)": "0.0001", "n with probs": "18"}. Runtime diagnostics {"slice": "mcpmark_tool_selection", "p50 ms": "683", "p95 ms": "3488", "p99 ms": "3488", "mean input tokens": "402", "total input tokens": "7,245", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 18 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · mcpmark_tool_selection · ECE (10 bins)	0.01 pp	Setup & source JevBench Choice slices: mcpmark_tool_selection · ECE (10 bins) Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "mcpmark_tool_selection", "n": "18", "gold classes": "5", "accuracy": "100.0%", "macro-F1": "1.0000", "Brier": "0.0000", "NLL": "0.0001", "zero-prob on gold": "0", "ECE (10 bins)": "0.0001", "n with probs": "18"}. Runtime diagnostics {"slice": "mcpmark_tool_selection", "p50 ms": "683", "p95 ms": "3488", "p99 ms": "3488", "mean input tokens": "402", "total input tokens": "7,245", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 18 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · medmcqa · accuracy	72.60%	Setup & source JevBench Choice slices: medmcqa · accuracy Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "medmcqa", "n": "1000", "gold classes": "4", "accuracy": "72.6%", "macro-F1": "0.7266", "Brier": "0.3953", "NLL": "0.7898", "zero-prob on gold": "0", "ECE (10 bins)": "0.1374", "n with probs": "1000"}. Runtime diagnostics {"slice": "medmcqa", "p50 ms": "590", "p95 ms": "1329", "p99 ms": "1806", "mean input tokens": "141", "total input tokens": "141,057", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 1000 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · medmcqa · macro-F1	72.66%	Setup & source JevBench Choice slices: medmcqa · macro-F1 Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "medmcqa", "n": "1000", "gold classes": "4", "accuracy": "72.6%", "macro-F1": "0.7266", "Brier": "0.3953", "NLL": "0.7898", "zero-prob on gold": "0", "ECE (10 bins)": "0.1374", "n with probs": "1000"}. Runtime diagnostics {"slice": "medmcqa", "p50 ms": "590", "p95 ms": "1329", "p99 ms": "1806", "mean input tokens": "141", "total input tokens": "141,057", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 1000 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · medmcqa · Brier	0.3953 Brier	Setup & source JevBench Choice slices: medmcqa · Brier Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "medmcqa", "n": "1000", "gold classes": "4", "accuracy": "72.6%", "macro-F1": "0.7266", "Brier": "0.3953", "NLL": "0.7898", "zero-prob on gold": "0", "ECE (10 bins)": "0.1374", "n with probs": "1000"}. Runtime diagnostics {"slice": "medmcqa", "p50 ms": "590", "p95 ms": "1329", "p99 ms": "1806", "mean input tokens": "141", "total input tokens": "141,057", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 1000 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · medmcqa · NLL	0.7898 NLL	Setup & source JevBench Choice slices: medmcqa · NLL Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "medmcqa", "n": "1000", "gold classes": "4", "accuracy": "72.6%", "macro-F1": "0.7266", "Brier": "0.3953", "NLL": "0.7898", "zero-prob on gold": "0", "ECE (10 bins)": "0.1374", "n with probs": "1000"}. Runtime diagnostics {"slice": "medmcqa", "p50 ms": "590", "p95 ms": "1329", "p99 ms": "1806", "mean input tokens": "141", "total input tokens": "141,057", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 1000 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · medmcqa · ECE (10 bins)	13.74 pp	Setup & source JevBench Choice slices: medmcqa · ECE (10 bins) Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "medmcqa", "n": "1000", "gold classes": "4", "accuracy": "72.6%", "macro-F1": "0.7266", "Brier": "0.3953", "NLL": "0.7898", "zero-prob on gold": "0", "ECE (10 bins)": "0.1374", "n with probs": "1000"}. Runtime diagnostics {"slice": "medmcqa", "p50 ms": "590", "p95 ms": "1329", "p99 ms": "1806", "mean input tokens": "141", "total input tokens": "141,057", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 1000 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · medqa_usmle · accuracy	84.90%	Setup & source JevBench Choice slices: medqa_usmle · accuracy Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "medqa_usmle", "n": "1000", "gold classes": "4", "accuracy": "84.9%", "macro-F1": "0.8486", "Brier": "0.2433", "NLL": "0.4988", "zero-prob on gold": "0", "ECE (10 bins)": "0.0816", "n with probs": "1000"}. Runtime diagnostics {"slice": "medqa_usmle", "p50 ms": "596", "p95 ms": "1546", "p99 ms": "2754", "mean input tokens": "301", "total input tokens": "301,460", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 1000 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · medqa_usmle · macro-F1	84.86%	Setup & source JevBench Choice slices: medqa_usmle · macro-F1 Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "medqa_usmle", "n": "1000", "gold classes": "4", "accuracy": "84.9%", "macro-F1": "0.8486", "Brier": "0.2433", "NLL": "0.4988", "zero-prob on gold": "0", "ECE (10 bins)": "0.0816", "n with probs": "1000"}. Runtime diagnostics {"slice": "medqa_usmle", "p50 ms": "596", "p95 ms": "1546", "p99 ms": "2754", "mean input tokens": "301", "total input tokens": "301,460", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 1000 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · medqa_usmle · Brier	0.2433 Brier	Setup & source JevBench Choice slices: medqa_usmle · Brier Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "medqa_usmle", "n": "1000", "gold classes": "4", "accuracy": "84.9%", "macro-F1": "0.8486", "Brier": "0.2433", "NLL": "0.4988", "zero-prob on gold": "0", "ECE (10 bins)": "0.0816", "n with probs": "1000"}. Runtime diagnostics {"slice": "medqa_usmle", "p50 ms": "596", "p95 ms": "1546", "p99 ms": "2754", "mean input tokens": "301", "total input tokens": "301,460", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 1000 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · medqa_usmle · NLL	0.4988 NLL	Setup & source JevBench Choice slices: medqa_usmle · NLL Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "medqa_usmle", "n": "1000", "gold classes": "4", "accuracy": "84.9%", "macro-F1": "0.8486", "Brier": "0.2433", "NLL": "0.4988", "zero-prob on gold": "0", "ECE (10 bins)": "0.0816", "n with probs": "1000"}. Runtime diagnostics {"slice": "medqa_usmle", "p50 ms": "596", "p95 ms": "1546", "p99 ms": "2754", "mean input tokens": "301", "total input tokens": "301,460", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 1000 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · medqa_usmle · ECE (10 bins)	8.16 pp	Setup & source JevBench Choice slices: medqa_usmle · ECE (10 bins) Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "medqa_usmle", "n": "1000", "gold classes": "4", "accuracy": "84.9%", "macro-F1": "0.8486", "Brier": "0.2433", "NLL": "0.4988", "zero-prob on gold": "0", "ECE (10 bins)": "0.0816", "n with probs": "1000"}. Runtime diagnostics {"slice": "medqa_usmle", "p50 ms": "596", "p95 ms": "1546", "p99 ms": "2754", "mean input tokens": "301", "total input tokens": "301,460", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 1000 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · mmlu_pro · accuracy	64.50%	Setup & source JevBench Choice slices: mmlu_pro · accuracy Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "mmlu_pro", "n": "1000", "gold classes": "10", "accuracy": "64.5%", "macro-F1": "0.6419", "Brier": "0.5369", "NLL": "1.3122", "zero-prob on gold": "0", "ECE (10 bins)": "0.1796", "n with probs": "1000"}. Runtime diagnostics {"slice": "mmlu_pro", "p50 ms": "625", "p95 ms": "1712", "p99 ms": "2715", "mean input tokens": "333", "total input tokens": "332,557", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 1000 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · mmlu_pro · macro-F1	64.19%	Setup & source JevBench Choice slices: mmlu_pro · macro-F1 Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "mmlu_pro", "n": "1000", "gold classes": "10", "accuracy": "64.5%", "macro-F1": "0.6419", "Brier": "0.5369", "NLL": "1.3122", "zero-prob on gold": "0", "ECE (10 bins)": "0.1796", "n with probs": "1000"}. Runtime diagnostics {"slice": "mmlu_pro", "p50 ms": "625", "p95 ms": "1712", "p99 ms": "2715", "mean input tokens": "333", "total input tokens": "332,557", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 1000 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · mmlu_pro · Brier	0.5369 Brier	Setup & source JevBench Choice slices: mmlu_pro · Brier Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "mmlu_pro", "n": "1000", "gold classes": "10", "accuracy": "64.5%", "macro-F1": "0.6419", "Brier": "0.5369", "NLL": "1.3122", "zero-prob on gold": "0", "ECE (10 bins)": "0.1796", "n with probs": "1000"}. Runtime diagnostics {"slice": "mmlu_pro", "p50 ms": "625", "p95 ms": "1712", "p99 ms": "2715", "mean input tokens": "333", "total input tokens": "332,557", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 1000 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · mmlu_pro · NLL	1.3122 NLL	Setup & source JevBench Choice slices: mmlu_pro · NLL Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "mmlu_pro", "n": "1000", "gold classes": "10", "accuracy": "64.5%", "macro-F1": "0.6419", "Brier": "0.5369", "NLL": "1.3122", "zero-prob on gold": "0", "ECE (10 bins)": "0.1796", "n with probs": "1000"}. Runtime diagnostics {"slice": "mmlu_pro", "p50 ms": "625", "p95 ms": "1712", "p99 ms": "2715", "mean input tokens": "333", "total input tokens": "332,557", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 1000 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · mmlu_pro · ECE (10 bins)	17.96 pp	Setup & source JevBench Choice slices: mmlu_pro · ECE (10 bins) Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "mmlu_pro", "n": "1000", "gold classes": "10", "accuracy": "64.5%", "macro-F1": "0.6419", "Brier": "0.5369", "NLL": "1.3122", "zero-prob on gold": "0", "ECE (10 bins)": "0.1796", "n with probs": "1000"}. Runtime diagnostics {"slice": "mmlu_pro", "p50 ms": "625", "p95 ms": "1712", "p99 ms": "2715", "mean input tokens": "333", "total input tokens": "332,557", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 1000 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · pubmedqa · accuracy	65.60%	Setup & source JevBench Choice slices: pubmedqa · accuracy Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "pubmedqa", "n": "500", "gold classes": "3", "accuracy": "65.6%", "macro-F1": "0.6412", "Brier": "0.5671", "NLL": "1.0990", "zero-prob on gold": "0", "ECE (10 bins)": "0.2184", "n with probs": "500"}. Runtime diagnostics {"slice": "pubmedqa", "p50 ms": "1063", "p95 ms": "2672", "p99 ms": "3697", "mean input tokens": "457", "total input tokens": "228,509", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 500 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · pubmedqa · macro-F1	64.12%	Setup & source JevBench Choice slices: pubmedqa · macro-F1 Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "pubmedqa", "n": "500", "gold classes": "3", "accuracy": "65.6%", "macro-F1": "0.6412", "Brier": "0.5671", "NLL": "1.0990", "zero-prob on gold": "0", "ECE (10 bins)": "0.2184", "n with probs": "500"}. Runtime diagnostics {"slice": "pubmedqa", "p50 ms": "1063", "p95 ms": "2672", "p99 ms": "3697", "mean input tokens": "457", "total input tokens": "228,509", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 500 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · pubmedqa · Brier	0.5671 Brier	Setup & source JevBench Choice slices: pubmedqa · Brier Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "pubmedqa", "n": "500", "gold classes": "3", "accuracy": "65.6%", "macro-F1": "0.6412", "Brier": "0.5671", "NLL": "1.0990", "zero-prob on gold": "0", "ECE (10 bins)": "0.2184", "n with probs": "500"}. Runtime diagnostics {"slice": "pubmedqa", "p50 ms": "1063", "p95 ms": "2672", "p99 ms": "3697", "mean input tokens": "457", "total input tokens": "228,509", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 500 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · pubmedqa · NLL	1.0990 NLL	Setup & source JevBench Choice slices: pubmedqa · NLL Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "pubmedqa", "n": "500", "gold classes": "3", "accuracy": "65.6%", "macro-F1": "0.6412", "Brier": "0.5671", "NLL": "1.0990", "zero-prob on gold": "0", "ECE (10 bins)": "0.2184", "n with probs": "500"}. Runtime diagnostics {"slice": "pubmedqa", "p50 ms": "1063", "p95 ms": "2672", "p99 ms": "3697", "mean input tokens": "457", "total input tokens": "228,509", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 500 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · pubmedqa · ECE (10 bins)	21.84 pp	Setup & source JevBench Choice slices: pubmedqa · ECE (10 bins) Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "pubmedqa", "n": "500", "gold classes": "3", "accuracy": "65.6%", "macro-F1": "0.6412", "Brier": "0.5671", "NLL": "1.0990", "zero-prob on gold": "0", "ECE (10 bins)": "0.2184", "n with probs": "500"}. Runtime diagnostics {"slice": "pubmedqa", "p50 ms": "1063", "p95 ms": "2672", "p99 ms": "3697", "mean input tokens": "457", "total input tokens": "228,509", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 500 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · rewardbench2 · accuracy	79.30%	Setup & source JevBench Choice slices: rewardbench2 · accuracy Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "rewardbench2", "n": "1200", "gold classes": "4", "accuracy": "79.3%", "macro-F1": "0.7932", "Brier": "0.3020", "NLL": "0.6393", "zero-prob on gold": "0", "ECE (10 bins)": "0.0529", "n with probs": "1200"}. Runtime diagnostics {"slice": "rewardbench2", "p50 ms": "896", "p95 ms": "2057", "p99 ms": "2813", "mean input tokens": "1777", "total input tokens": "2,132,815", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 1200 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · rewardbench2 · macro-F1	79.32%	Setup & source JevBench Choice slices: rewardbench2 · macro-F1 Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "rewardbench2", "n": "1200", "gold classes": "4", "accuracy": "79.3%", "macro-F1": "0.7932", "Brier": "0.3020", "NLL": "0.6393", "zero-prob on gold": "0", "ECE (10 bins)": "0.0529", "n with probs": "1200"}. Runtime diagnostics {"slice": "rewardbench2", "p50 ms": "896", "p95 ms": "2057", "p99 ms": "2813", "mean input tokens": "1777", "total input tokens": "2,132,815", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 1200 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · rewardbench2 · Brier	0.3020 Brier	Setup & source JevBench Choice slices: rewardbench2 · Brier Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "rewardbench2", "n": "1200", "gold classes": "4", "accuracy": "79.3%", "macro-F1": "0.7932", "Brier": "0.3020", "NLL": "0.6393", "zero-prob on gold": "0", "ECE (10 bins)": "0.0529", "n with probs": "1200"}. Runtime diagnostics {"slice": "rewardbench2", "p50 ms": "896", "p95 ms": "2057", "p99 ms": "2813", "mean input tokens": "1777", "total input tokens": "2,132,815", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 1200 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · rewardbench2 · NLL	0.6393 NLL	Setup & source JevBench Choice slices: rewardbench2 · NLL Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "rewardbench2", "n": "1200", "gold classes": "4", "accuracy": "79.3%", "macro-F1": "0.7932", "Brier": "0.3020", "NLL": "0.6393", "zero-prob on gold": "0", "ECE (10 bins)": "0.0529", "n with probs": "1200"}. Runtime diagnostics {"slice": "rewardbench2", "p50 ms": "896", "p95 ms": "2057", "p99 ms": "2813", "mean input tokens": "1777", "total input tokens": "2,132,815", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 1200 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · rewardbench2 · ECE (10 bins)	5.29 pp	Setup & source JevBench Choice slices: rewardbench2 · ECE (10 bins) Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "rewardbench2", "n": "1200", "gold classes": "4", "accuracy": "79.3%", "macro-F1": "0.7932", "Brier": "0.3020", "NLL": "0.6393", "zero-prob on gold": "0", "ECE (10 bins)": "0.0529", "n with probs": "1200"}. Runtime diagnostics {"slice": "rewardbench2", "p50 ms": "896", "p95 ms": "2057", "p99 ms": "2813", "mean input tokens": "1777", "total input tokens": "2,132,815", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 1200 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · scienceqa_text · accuracy	94.20%	Setup & source JevBench Choice slices: scienceqa_text · accuracy Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "scienceqa_text", "n": "1000", "gold classes": "4", "accuracy": "94.2%", "macro-F1": "0.9665", "Brier": "0.0935", "NLL": "0.1716", "zero-prob on gold": "0", "ECE (10 bins)": "0.0410", "n with probs": "1000"}. Runtime diagnostics {"slice": "scienceqa_text", "p50 ms": "896", "p95 ms": "2274", "p99 ms": "2831", "mean input tokens": "129", "total input tokens": "129,244", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 1000 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · scienceqa_text · macro-F1	96.65%	Setup & source JevBench Choice slices: scienceqa_text · macro-F1 Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "scienceqa_text", "n": "1000", "gold classes": "4", "accuracy": "94.2%", "macro-F1": "0.9665", "Brier": "0.0935", "NLL": "0.1716", "zero-prob on gold": "0", "ECE (10 bins)": "0.0410", "n with probs": "1000"}. Runtime diagnostics {"slice": "scienceqa_text", "p50 ms": "896", "p95 ms": "2274", "p99 ms": "2831", "mean input tokens": "129", "total input tokens": "129,244", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 1000 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · scienceqa_text · Brier	0.0935 Brier	Setup & source JevBench Choice slices: scienceqa_text · Brier Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "scienceqa_text", "n": "1000", "gold classes": "4", "accuracy": "94.2%", "macro-F1": "0.9665", "Brier": "0.0935", "NLL": "0.1716", "zero-prob on gold": "0", "ECE (10 bins)": "0.0410", "n with probs": "1000"}. Runtime diagnostics {"slice": "scienceqa_text", "p50 ms": "896", "p95 ms": "2274", "p99 ms": "2831", "mean input tokens": "129", "total input tokens": "129,244", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 1000 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · scienceqa_text · NLL	0.1716 NLL	Setup & source JevBench Choice slices: scienceqa_text · NLL Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "scienceqa_text", "n": "1000", "gold classes": "4", "accuracy": "94.2%", "macro-F1": "0.9665", "Brier": "0.0935", "NLL": "0.1716", "zero-prob on gold": "0", "ECE (10 bins)": "0.0410", "n with probs": "1000"}. Runtime diagnostics {"slice": "scienceqa_text", "p50 ms": "896", "p95 ms": "2274", "p99 ms": "2831", "mean input tokens": "129", "total input tokens": "129,244", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 1000 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · scienceqa_text · ECE (10 bins)	4.10 pp	Setup & source JevBench Choice slices: scienceqa_text · ECE (10 bins) Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "scienceqa_text", "n": "1000", "gold classes": "4", "accuracy": "94.2%", "macro-F1": "0.9665", "Brier": "0.0935", "NLL": "0.1716", "zero-prob on gold": "0", "ECE (10 bins)": "0.0410", "n with probs": "1000"}. Runtime diagnostics {"slice": "scienceqa_text", "p50 ms": "896", "p95 ms": "2274", "p99 ms": "2831", "mean input tokens": "129", "total input tokens": "129,244", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 1000 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · aegis2 · acc @0.5	84.00%	Setup & source JevBench Binary slices: aegis2 · acc @0.5 Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "aegis2", "n": "500", "acc @0.5": "84.0%", "acc if flipped": "16.0%", "positive recall": "80.4%", "FPR": "12.4%", "AUROC": "0.9279", "Brier": "0.1275", "ECE": "0.1109", "base rate": "50.0%"}. Runtime diagnostics {"slice": "aegis2", "p50 ms": "623", "p95 ms": "5289", "p99 ms": "6059", "mean input tokens": "180", "total input tokens": "90,031", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 500 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · aegis2 · acc if flipped	16.00%	Setup & source JevBench Binary slices: aegis2 · acc if flipped Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "aegis2", "n": "500", "acc @0.5": "84.0%", "acc if flipped": "16.0%", "positive recall": "80.4%", "FPR": "12.4%", "AUROC": "0.9279", "Brier": "0.1275", "ECE": "0.1109", "base rate": "50.0%"}. Runtime diagnostics {"slice": "aegis2", "p50 ms": "623", "p95 ms": "5289", "p99 ms": "6059", "mean input tokens": "180", "total input tokens": "90,031", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 500 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · aegis2 · positive recall	80.40%	Setup & source JevBench Binary slices: aegis2 · positive recall Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "aegis2", "n": "500", "acc @0.5": "84.0%", "acc if flipped": "16.0%", "positive recall": "80.4%", "FPR": "12.4%", "AUROC": "0.9279", "Brier": "0.1275", "ECE": "0.1109", "base rate": "50.0%"}. Runtime diagnostics {"slice": "aegis2", "p50 ms": "623", "p95 ms": "5289", "p99 ms": "6059", "mean input tokens": "180", "total input tokens": "90,031", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 500 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · aegis2 · FPR	12.40%	Setup & source JevBench Binary slices: aegis2 · FPR Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "aegis2", "n": "500", "acc @0.5": "84.0%", "acc if flipped": "16.0%", "positive recall": "80.4%", "FPR": "12.4%", "AUROC": "0.9279", "Brier": "0.1275", "ECE": "0.1109", "base rate": "50.0%"}. Runtime diagnostics {"slice": "aegis2", "p50 ms": "623", "p95 ms": "5289", "p99 ms": "6059", "mean input tokens": "180", "total input tokens": "90,031", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 500 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · aegis2 · AUROC	92.79%	Setup & source JevBench Binary slices: aegis2 · AUROC Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "aegis2", "n": "500", "acc @0.5": "84.0%", "acc if flipped": "16.0%", "positive recall": "80.4%", "FPR": "12.4%", "AUROC": "0.9279", "Brier": "0.1275", "ECE": "0.1109", "base rate": "50.0%"}. Runtime diagnostics {"slice": "aegis2", "p50 ms": "623", "p95 ms": "5289", "p99 ms": "6059", "mean input tokens": "180", "total input tokens": "90,031", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 500 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · aegis2 · Brier	0.1275 Brier	Setup & source JevBench Binary slices: aegis2 · Brier Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "aegis2", "n": "500", "acc @0.5": "84.0%", "acc if flipped": "16.0%", "positive recall": "80.4%", "FPR": "12.4%", "AUROC": "0.9279", "Brier": "0.1275", "ECE": "0.1109", "base rate": "50.0%"}. Runtime diagnostics {"slice": "aegis2", "p50 ms": "623", "p95 ms": "5289", "p99 ms": "6059", "mean input tokens": "180", "total input tokens": "90,031", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 500 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · aegis2 · ECE	11.09 pp	Setup & source JevBench Binary slices: aegis2 · ECE Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "aegis2", "n": "500", "acc @0.5": "84.0%", "acc if flipped": "16.0%", "positive recall": "80.4%", "FPR": "12.4%", "AUROC": "0.9279", "Brier": "0.1275", "ECE": "0.1109", "base rate": "50.0%"}. Runtime diagnostics {"slice": "aegis2", "p50 ms": "623", "p95 ms": "5289", "p99 ms": "6059", "mean input tokens": "180", "total input tokens": "90,031", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 500 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · aegis2_response · acc @0.5	77.20%	Setup & source JevBench Binary slices: aegis2_response · acc @0.5 Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "aegis2_response", "n": "500", "acc @0.5": "77.2%", "acc if flipped": "22.8%", "positive recall": "62.4%", "FPR": "8.0%", "AUROC": "0.8876", "Brier": "0.1888", "ECE": "0.1655", "base rate": "50.0%"}. Runtime diagnostics {"slice": "aegis2_response", "p50 ms": "564", "p95 ms": "1376", "p99 ms": "1831", "mean input tokens": "282", "total input tokens": "140,800", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 500 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · aegis2_response · acc if flipped	22.80%	Setup & source JevBench Binary slices: aegis2_response · acc if flipped Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "aegis2_response", "n": "500", "acc @0.5": "77.2%", "acc if flipped": "22.8%", "positive recall": "62.4%", "FPR": "8.0%", "AUROC": "0.8876", "Brier": "0.1888", "ECE": "0.1655", "base rate": "50.0%"}. Runtime diagnostics {"slice": "aegis2_response", "p50 ms": "564", "p95 ms": "1376", "p99 ms": "1831", "mean input tokens": "282", "total input tokens": "140,800", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 500 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · aegis2_response · positive recall	62.40%	Setup & source JevBench Binary slices: aegis2_response · positive recall Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "aegis2_response", "n": "500", "acc @0.5": "77.2%", "acc if flipped": "22.8%", "positive recall": "62.4%", "FPR": "8.0%", "AUROC": "0.8876", "Brier": "0.1888", "ECE": "0.1655", "base rate": "50.0%"}. Runtime diagnostics {"slice": "aegis2_response", "p50 ms": "564", "p95 ms": "1376", "p99 ms": "1831", "mean input tokens": "282", "total input tokens": "140,800", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 500 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · aegis2_response · FPR	8.00%	Setup & source JevBench Binary slices: aegis2_response · FPR Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "aegis2_response", "n": "500", "acc @0.5": "77.2%", "acc if flipped": "22.8%", "positive recall": "62.4%", "FPR": "8.0%", "AUROC": "0.8876", "Brier": "0.1888", "ECE": "0.1655", "base rate": "50.0%"}. Runtime diagnostics {"slice": "aegis2_response", "p50 ms": "564", "p95 ms": "1376", "p99 ms": "1831", "mean input tokens": "282", "total input tokens": "140,800", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 500 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · aegis2_response · AUROC	88.76%	Setup & source JevBench Binary slices: aegis2_response · AUROC Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "aegis2_response", "n": "500", "acc @0.5": "77.2%", "acc if flipped": "22.8%", "positive recall": "62.4%", "FPR": "8.0%", "AUROC": "0.8876", "Brier": "0.1888", "ECE": "0.1655", "base rate": "50.0%"}. Runtime diagnostics {"slice": "aegis2_response", "p50 ms": "564", "p95 ms": "1376", "p99 ms": "1831", "mean input tokens": "282", "total input tokens": "140,800", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 500 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · aegis2_response · Brier	0.1888 Brier	Setup & source JevBench Binary slices: aegis2_response · Brier Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "aegis2_response", "n": "500", "acc @0.5": "77.2%", "acc if flipped": "22.8%", "positive recall": "62.4%", "FPR": "8.0%", "AUROC": "0.8876", "Brier": "0.1888", "ECE": "0.1655", "base rate": "50.0%"}. Runtime diagnostics {"slice": "aegis2_response", "p50 ms": "564", "p95 ms": "1376", "p99 ms": "1831", "mean input tokens": "282", "total input tokens": "140,800", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 500 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · aegis2_response · ECE	16.55 pp	Setup & source JevBench Binary slices: aegis2_response · ECE Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "aegis2_response", "n": "500", "acc @0.5": "77.2%", "acc if flipped": "22.8%", "positive recall": "62.4%", "FPR": "8.0%", "AUROC": "0.8876", "Brier": "0.1888", "ECE": "0.1655", "base rate": "50.0%"}. Runtime diagnostics {"slice": "aegis2_response", "p50 ms": "564", "p95 ms": "1376", "p99 ms": "1831", "mean input tokens": "282", "total input tokens": "140,800", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 500 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · atbench500 · acc @0.5	68.80%	Setup & source JevBench Binary slices: atbench500 · acc @0.5 Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "atbench500", "n": "500", "acc @0.5": "68.8%", "acc if flipped": "31.2%", "positive recall": "37.6%", "FPR": "0.0%", "AUROC": "0.9932", "Brier": "0.2794", "ECE": "0.2826", "base rate": "50.0%"}. Runtime diagnostics {"slice": "atbench500", "p50 ms": "851", "p95 ms": "1790", "p99 ms": "2262", "mean input tokens": "1895", "total input tokens": "947,364", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 500 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · atbench500 · acc if flipped	31.20%	Setup & source JevBench Binary slices: atbench500 · acc if flipped Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "atbench500", "n": "500", "acc @0.5": "68.8%", "acc if flipped": "31.2%", "positive recall": "37.6%", "FPR": "0.0%", "AUROC": "0.9932", "Brier": "0.2794", "ECE": "0.2826", "base rate": "50.0%"}. Runtime diagnostics {"slice": "atbench500", "p50 ms": "851", "p95 ms": "1790", "p99 ms": "2262", "mean input tokens": "1895", "total input tokens": "947,364", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 500 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · atbench500 · positive recall	37.60%	Setup & source JevBench Binary slices: atbench500 · positive recall Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "atbench500", "n": "500", "acc @0.5": "68.8%", "acc if flipped": "31.2%", "positive recall": "37.6%", "FPR": "0.0%", "AUROC": "0.9932", "Brier": "0.2794", "ECE": "0.2826", "base rate": "50.0%"}. Runtime diagnostics {"slice": "atbench500", "p50 ms": "851", "p95 ms": "1790", "p99 ms": "2262", "mean input tokens": "1895", "total input tokens": "947,364", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 500 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · atbench500 · FPR	0.00%	Setup & source JevBench Binary slices: atbench500 · FPR Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "atbench500", "n": "500", "acc @0.5": "68.8%", "acc if flipped": "31.2%", "positive recall": "37.6%", "FPR": "0.0%", "AUROC": "0.9932", "Brier": "0.2794", "ECE": "0.2826", "base rate": "50.0%"}. Runtime diagnostics {"slice": "atbench500", "p50 ms": "851", "p95 ms": "1790", "p99 ms": "2262", "mean input tokens": "1895", "total input tokens": "947,364", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 500 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · atbench500 · AUROC	99.32%	Setup & source JevBench Binary slices: atbench500 · AUROC Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "atbench500", "n": "500", "acc @0.5": "68.8%", "acc if flipped": "31.2%", "positive recall": "37.6%", "FPR": "0.0%", "AUROC": "0.9932", "Brier": "0.2794", "ECE": "0.2826", "base rate": "50.0%"}. Runtime diagnostics {"slice": "atbench500", "p50 ms": "851", "p95 ms": "1790", "p99 ms": "2262", "mean input tokens": "1895", "total input tokens": "947,364", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 500 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · atbench500 · Brier	0.2794 Brier	Setup & source JevBench Binary slices: atbench500 · Brier Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "atbench500", "n": "500", "acc @0.5": "68.8%", "acc if flipped": "31.2%", "positive recall": "37.6%", "FPR": "0.0%", "AUROC": "0.9932", "Brier": "0.2794", "ECE": "0.2826", "base rate": "50.0%"}. Runtime diagnostics {"slice": "atbench500", "p50 ms": "851", "p95 ms": "1790", "p99 ms": "2262", "mean input tokens": "1895", "total input tokens": "947,364", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 500 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · atbench500 · ECE	28.26 pp	Setup & source JevBench Binary slices: atbench500 · ECE Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "atbench500", "n": "500", "acc @0.5": "68.8%", "acc if flipped": "31.2%", "positive recall": "37.6%", "FPR": "0.0%", "AUROC": "0.9932", "Brier": "0.2794", "ECE": "0.2826", "base rate": "50.0%"}. Runtime diagnostics {"slice": "atbench500", "p50 ms": "851", "p95 ms": "1790", "p99 ms": "2262", "mean input tokens": "1895", "total input tokens": "947,364", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 500 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · jailbreak_classification · acc @0.5	96.80%	Setup & source JevBench Binary slices: jailbreak_classification · acc @0.5 Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "jailbreak_classification", "n": "400", "acc @0.5": "96.8%", "acc if flipped": "3.2%", "positive recall": "90.8%", "FPR": "0.0%", "AUROC": "0.9930", "Brier": "0.0301", "ECE": "0.0261", "base rate": "35.2%"}. Runtime diagnostics {"slice": "jailbreak_classification", "p50 ms": "577", "p95 ms": "1288", "p99 ms": "1876", "mean input tokens": "388", "total input tokens": "155,110", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 400 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · jailbreak_classification · acc if flipped	3.20%	Setup & source JevBench Binary slices: jailbreak_classification · acc if flipped Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "jailbreak_classification", "n": "400", "acc @0.5": "96.8%", "acc if flipped": "3.2%", "positive recall": "90.8%", "FPR": "0.0%", "AUROC": "0.9930", "Brier": "0.0301", "ECE": "0.0261", "base rate": "35.2%"}. Runtime diagnostics {"slice": "jailbreak_classification", "p50 ms": "577", "p95 ms": "1288", "p99 ms": "1876", "mean input tokens": "388", "total input tokens": "155,110", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 400 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · jailbreak_classification · positive recall	90.80%	Setup & source JevBench Binary slices: jailbreak_classification · positive recall Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "jailbreak_classification", "n": "400", "acc @0.5": "96.8%", "acc if flipped": "3.2%", "positive recall": "90.8%", "FPR": "0.0%", "AUROC": "0.9930", "Brier": "0.0301", "ECE": "0.0261", "base rate": "35.2%"}. Runtime diagnostics {"slice": "jailbreak_classification", "p50 ms": "577", "p95 ms": "1288", "p99 ms": "1876", "mean input tokens": "388", "total input tokens": "155,110", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 400 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · jailbreak_classification · FPR	0.00%	Setup & source JevBench Binary slices: jailbreak_classification · FPR Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "jailbreak_classification", "n": "400", "acc @0.5": "96.8%", "acc if flipped": "3.2%", "positive recall": "90.8%", "FPR": "0.0%", "AUROC": "0.9930", "Brier": "0.0301", "ECE": "0.0261", "base rate": "35.2%"}. Runtime diagnostics {"slice": "jailbreak_classification", "p50 ms": "577", "p95 ms": "1288", "p99 ms": "1876", "mean input tokens": "388", "total input tokens": "155,110", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 400 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · jailbreak_classification · AUROC	99.30%	Setup & source JevBench Binary slices: jailbreak_classification · AUROC Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "jailbreak_classification", "n": "400", "acc @0.5": "96.8%", "acc if flipped": "3.2%", "positive recall": "90.8%", "FPR": "0.0%", "AUROC": "0.9930", "Brier": "0.0301", "ECE": "0.0261", "base rate": "35.2%"}. Runtime diagnostics {"slice": "jailbreak_classification", "p50 ms": "577", "p95 ms": "1288", "p99 ms": "1876", "mean input tokens": "388", "total input tokens": "155,110", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 400 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · jailbreak_classification · Brier	0.0301 Brier	Setup & source JevBench Binary slices: jailbreak_classification · Brier Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "jailbreak_classification", "n": "400", "acc @0.5": "96.8%", "acc if flipped": "3.2%", "positive recall": "90.8%", "FPR": "0.0%", "AUROC": "0.9930", "Brier": "0.0301", "ECE": "0.0261", "base rate": "35.2%"}. Runtime diagnostics {"slice": "jailbreak_classification", "p50 ms": "577", "p95 ms": "1288", "p99 ms": "1876", "mean input tokens": "388", "total input tokens": "155,110", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 400 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · jailbreak_classification · ECE	2.61 pp	Setup & source JevBench Binary slices: jailbreak_classification · ECE Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "jailbreak_classification", "n": "400", "acc @0.5": "96.8%", "acc if flipped": "3.2%", "positive recall": "90.8%", "FPR": "0.0%", "AUROC": "0.9930", "Brier": "0.0301", "ECE": "0.0261", "base rate": "35.2%"}. Runtime diagnostics {"slice": "jailbreak_classification", "p50 ms": "577", "p95 ms": "1288", "p99 ms": "1876", "mean input tokens": "388", "total input tokens": "155,110", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 400 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · prompt_injections · acc @0.5	81.00%	Setup & source JevBench Binary slices: prompt_injections · acc @0.5 Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "prompt_injections", "n": "116", "acc @0.5": "81.0%", "acc if flipped": "19.0%", "positive recall": "63.3%", "FPR": "0.0%", "AUROC": "0.9851", "Brier": "0.1570", "ECE": "0.1566", "base rate": "51.7%"}. Runtime diagnostics {"slice": "prompt_injections", "p50 ms": "996", "p95 ms": "2094", "p99 ms": "3122", "mean input tokens": "171", "total input tokens": "19,796", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 116 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · prompt_injections · acc if flipped	19.00%	Setup & source JevBench Binary slices: prompt_injections · acc if flipped Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "prompt_injections", "n": "116", "acc @0.5": "81.0%", "acc if flipped": "19.0%", "positive recall": "63.3%", "FPR": "0.0%", "AUROC": "0.9851", "Brier": "0.1570", "ECE": "0.1566", "base rate": "51.7%"}. Runtime diagnostics {"slice": "prompt_injections", "p50 ms": "996", "p95 ms": "2094", "p99 ms": "3122", "mean input tokens": "171", "total input tokens": "19,796", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 116 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · prompt_injections · positive recall	63.30%	Setup & source JevBench Binary slices: prompt_injections · positive recall Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "prompt_injections", "n": "116", "acc @0.5": "81.0%", "acc if flipped": "19.0%", "positive recall": "63.3%", "FPR": "0.0%", "AUROC": "0.9851", "Brier": "0.1570", "ECE": "0.1566", "base rate": "51.7%"}. Runtime diagnostics {"slice": "prompt_injections", "p50 ms": "996", "p95 ms": "2094", "p99 ms": "3122", "mean input tokens": "171", "total input tokens": "19,796", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 116 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · prompt_injections · FPR	0.00%	Setup & source JevBench Binary slices: prompt_injections · FPR Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "prompt_injections", "n": "116", "acc @0.5": "81.0%", "acc if flipped": "19.0%", "positive recall": "63.3%", "FPR": "0.0%", "AUROC": "0.9851", "Brier": "0.1570", "ECE": "0.1566", "base rate": "51.7%"}. Runtime diagnostics {"slice": "prompt_injections", "p50 ms": "996", "p95 ms": "2094", "p99 ms": "3122", "mean input tokens": "171", "total input tokens": "19,796", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 116 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · prompt_injections · AUROC	98.51%	Setup & source JevBench Binary slices: prompt_injections · AUROC Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "prompt_injections", "n": "116", "acc @0.5": "81.0%", "acc if flipped": "19.0%", "positive recall": "63.3%", "FPR": "0.0%", "AUROC": "0.9851", "Brier": "0.1570", "ECE": "0.1566", "base rate": "51.7%"}. Runtime diagnostics {"slice": "prompt_injections", "p50 ms": "996", "p95 ms": "2094", "p99 ms": "3122", "mean input tokens": "171", "total input tokens": "19,796", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 116 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · prompt_injections · Brier	0.1570 Brier	Setup & source JevBench Binary slices: prompt_injections · Brier Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "prompt_injections", "n": "116", "acc @0.5": "81.0%", "acc if flipped": "19.0%", "positive recall": "63.3%", "FPR": "0.0%", "AUROC": "0.9851", "Brier": "0.1570", "ECE": "0.1566", "base rate": "51.7%"}. Runtime diagnostics {"slice": "prompt_injections", "p50 ms": "996", "p95 ms": "2094", "p99 ms": "3122", "mean input tokens": "171", "total input tokens": "19,796", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 116 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · prompt_injections · ECE	15.66 pp	Setup & source JevBench Binary slices: prompt_injections · ECE Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "prompt_injections", "n": "116", "acc @0.5": "81.0%", "acc if flipped": "19.0%", "positive recall": "63.3%", "FPR": "0.0%", "AUROC": "0.9851", "Brier": "0.1570", "ECE": "0.1566", "base rate": "51.7%"}. Runtime diagnostics {"slice": "prompt_injections", "p50 ms": "996", "p95 ms": "2094", "p99 ms": "3122", "mean input tokens": "171", "total input tokens": "19,796", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 116 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · helpsteer2_helpfulness · exact (rounded)	35.60%	Setup & source JevBench Ordinal slices: helpsteer2_helpfulness · exact (rounded) Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "helpsteer2_helpfulness", "n": "500", "levels": "5", "exact (rounded)": "35.6%", "within 1 level": "78.6%", "MAE (expected level)": "0.947", "RPS": "0.1763", "Spearman": "0.555"}. Runtime diagnostics {"slice": "helpsteer2_helpfulness", "p50 ms": "924", "p95 ms": "6076", "p99 ms": "6843", "mean input tokens": "670", "total input tokens": "335,063", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 500 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · helpsteer2_helpfulness · within 1 level	78.60%	Setup & source JevBench Ordinal slices: helpsteer2_helpfulness · within 1 level Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "helpsteer2_helpfulness", "n": "500", "levels": "5", "exact (rounded)": "35.6%", "within 1 level": "78.6%", "MAE (expected level)": "0.947", "RPS": "0.1763", "Spearman": "0.555"}. Runtime diagnostics {"slice": "helpsteer2_helpfulness", "p50 ms": "924", "p95 ms": "6076", "p99 ms": "6843", "mean input tokens": "670", "total input tokens": "335,063", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 500 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · helpsteer2_helpfulness · MAE (expected level)	0.9470 MAE	Setup & source JevBench Ordinal slices: helpsteer2_helpfulness · MAE (expected level) Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "helpsteer2_helpfulness", "n": "500", "levels": "5", "exact (rounded)": "35.6%", "within 1 level": "78.6%", "MAE (expected level)": "0.947", "RPS": "0.1763", "Spearman": "0.555"}. Runtime diagnostics {"slice": "helpsteer2_helpfulness", "p50 ms": "924", "p95 ms": "6076", "p99 ms": "6843", "mean input tokens": "670", "total input tokens": "335,063", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 500 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · helpsteer2_helpfulness · RPS	0.1763 RPS	Setup & source JevBench Ordinal slices: helpsteer2_helpfulness · RPS Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "helpsteer2_helpfulness", "n": "500", "levels": "5", "exact (rounded)": "35.6%", "within 1 level": "78.6%", "MAE (expected level)": "0.947", "RPS": "0.1763", "Spearman": "0.555"}. Runtime diagnostics {"slice": "helpsteer2_helpfulness", "p50 ms": "924", "p95 ms": "6076", "p99 ms": "6843", "mean input tokens": "670", "total input tokens": "335,063", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 500 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · helpsteer2_helpfulness · Spearman	0.5550 coefficient	Setup & source JevBench Ordinal slices: helpsteer2_helpfulness · Spearman Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "helpsteer2_helpfulness", "n": "500", "levels": "5", "exact (rounded)": "35.6%", "within 1 level": "78.6%", "MAE (expected level)": "0.947", "RPS": "0.1763", "Spearman": "0.555"}. Runtime diagnostics {"slice": "helpsteer2_helpfulness", "p50 ms": "924", "p95 ms": "6076", "p99 ms": "6843", "mean input tokens": "670", "total input tokens": "335,063", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 500 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · nfcorpus_rank_graded · exact (rounded)	78.00%	Setup & source JevBench Ordinal slices: nfcorpus_rank_graded · exact (rounded) Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "nfcorpus_rank_graded", "n": "4924", "levels": "3", "exact (rounded)": "78.0%", "within 1 level": "97.1%", "MAE (expected level)": "0.278", "RPS": "0.1000", "Spearman": "0.471"}. Runtime diagnostics {"slice": "nfcorpus_rank_graded", "p50 ms": "582", "p95 ms": "1449", "p99 ms": "2023", "mean input tokens": "307", "total input tokens": "1,511,895", "retries": "1"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 4924 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · nfcorpus_rank_graded · within 1 level	97.10%	Setup & source JevBench Ordinal slices: nfcorpus_rank_graded · within 1 level Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "nfcorpus_rank_graded", "n": "4924", "levels": "3", "exact (rounded)": "78.0%", "within 1 level": "97.1%", "MAE (expected level)": "0.278", "RPS": "0.1000", "Spearman": "0.471"}. Runtime diagnostics {"slice": "nfcorpus_rank_graded", "p50 ms": "582", "p95 ms": "1449", "p99 ms": "2023", "mean input tokens": "307", "total input tokens": "1,511,895", "retries": "1"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 4924 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · nfcorpus_rank_graded · MAE (expected level)	0.2780 MAE	Setup & source JevBench Ordinal slices: nfcorpus_rank_graded · MAE (expected level) Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "nfcorpus_rank_graded", "n": "4924", "levels": "3", "exact (rounded)": "78.0%", "within 1 level": "97.1%", "MAE (expected level)": "0.278", "RPS": "0.1000", "Spearman": "0.471"}. Runtime diagnostics {"slice": "nfcorpus_rank_graded", "p50 ms": "582", "p95 ms": "1449", "p99 ms": "2023", "mean input tokens": "307", "total input tokens": "1,511,895", "retries": "1"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 4924 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · nfcorpus_rank_graded · RPS	0.1000 RPS	Setup & source JevBench Ordinal slices: nfcorpus_rank_graded · RPS Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "nfcorpus_rank_graded", "n": "4924", "levels": "3", "exact (rounded)": "78.0%", "within 1 level": "97.1%", "MAE (expected level)": "0.278", "RPS": "0.1000", "Spearman": "0.471"}. Runtime diagnostics {"slice": "nfcorpus_rank_graded", "p50 ms": "582", "p95 ms": "1449", "p99 ms": "2023", "mean input tokens": "307", "total input tokens": "1,511,895", "retries": "1"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 4924 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · nfcorpus_rank_graded · Spearman	0.4710 coefficient	Setup & source JevBench Ordinal slices: nfcorpus_rank_graded · Spearman Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "nfcorpus_rank_graded", "n": "4924", "levels": "3", "exact (rounded)": "78.0%", "within 1 level": "97.1%", "MAE (expected level)": "0.278", "RPS": "0.1000", "Spearman": "0.471"}. Runtime diagnostics {"slice": "nfcorpus_rank_graded", "p50 ms": "582", "p95 ms": "1449", "p99 ms": "2023", "mean input tokens": "307", "total input tokens": "1,511,895", "retries": "1"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 4924 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · sst5 · exact (rounded)	59.40%	Setup & source JevBench Ordinal slices: sst5 · exact (rounded) Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "sst5", "n": "500", "levels": "5", "exact (rounded)": "59.4%", "within 1 level": "97.6%", "MAE (expected level)": "0.477", "RPS": "0.0811", "Spearman": "0.890"}. Runtime diagnostics {"slice": "sst5", "p50 ms": "833", "p95 ms": "1986", "p99 ms": "2713", "mean input tokens": "139", "total input tokens": "69,251", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 500 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · sst5 · within 1 level	97.60%	Setup & source JevBench Ordinal slices: sst5 · within 1 level Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "sst5", "n": "500", "levels": "5", "exact (rounded)": "59.4%", "within 1 level": "97.6%", "MAE (expected level)": "0.477", "RPS": "0.0811", "Spearman": "0.890"}. Runtime diagnostics {"slice": "sst5", "p50 ms": "833", "p95 ms": "1986", "p99 ms": "2713", "mean input tokens": "139", "total input tokens": "69,251", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 500 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · sst5 · MAE (expected level)	0.4770 MAE	Setup & source JevBench Ordinal slices: sst5 · MAE (expected level) Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "sst5", "n": "500", "levels": "5", "exact (rounded)": "59.4%", "within 1 level": "97.6%", "MAE (expected level)": "0.477", "RPS": "0.0811", "Spearman": "0.890"}. Runtime diagnostics {"slice": "sst5", "p50 ms": "833", "p95 ms": "1986", "p99 ms": "2713", "mean input tokens": "139", "total input tokens": "69,251", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 500 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · sst5 · RPS	0.0811 RPS	Setup & source JevBench Ordinal slices: sst5 · RPS Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "sst5", "n": "500", "levels": "5", "exact (rounded)": "59.4%", "within 1 level": "97.6%", "MAE (expected level)": "0.477", "RPS": "0.0811", "Spearman": "0.890"}. Runtime diagnostics {"slice": "sst5", "p50 ms": "833", "p95 ms": "1986", "p99 ms": "2713", "mean input tokens": "139", "total input tokens": "69,251", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 500 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · sst5 · Spearman	0.8900 coefficient	Setup & source JevBench Ordinal slices: sst5 · Spearman Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "sst5", "n": "500", "levels": "5", "exact (rounded)": "59.4%", "within 1 level": "97.6%", "MAE (expected level)": "0.477", "RPS": "0.0811", "Spearman": "0.890"}. Runtime diagnostics {"slice": "sst5", "p50 ms": "833", "p95 ms": "1986", "p99 ms": "2713", "mean input tokens": "139", "total input tokens": "69,251", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 500 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · nfcorpus_rank_choice · nDCG@10	35.54%	Setup & source JevBench Ranking slices: nfcorpus_rank_choice · nDCG@10 Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "nfcorpus_rank_choice", "queries": "298", "nDCG@10": "0.3554", "MRR@10": "0.6219", "Recall@10": "0.1660", "BM25 nDCG@10": "0.3319", "ceiling nDCG@10": "0.4651", "shortlist recall": "19.1%"}. Runtime diagnostics {"slice": "nfcorpus_rank_choice", "p50 ms": "1131", "p95 ms": "6996", "p99 ms": "7522", "mean input tokens": "2380", "total input tokens": "709,228", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 298 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · nfcorpus_rank_choice · MRR@10	62.19%	Setup & source JevBench Ranking slices: nfcorpus_rank_choice · MRR@10 Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "nfcorpus_rank_choice", "queries": "298", "nDCG@10": "0.3554", "MRR@10": "0.6219", "Recall@10": "0.1660", "BM25 nDCG@10": "0.3319", "ceiling nDCG@10": "0.4651", "shortlist recall": "19.1%"}. Runtime diagnostics {"slice": "nfcorpus_rank_choice", "p50 ms": "1131", "p95 ms": "6996", "p99 ms": "7522", "mean input tokens": "2380", "total input tokens": "709,228", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 298 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · nfcorpus_rank_choice · Recall@10	16.60%	Setup & source JevBench Ranking slices: nfcorpus_rank_choice · Recall@10 Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "nfcorpus_rank_choice", "queries": "298", "nDCG@10": "0.3554", "MRR@10": "0.6219", "Recall@10": "0.1660", "BM25 nDCG@10": "0.3319", "ceiling nDCG@10": "0.4651", "shortlist recall": "19.1%"}. Runtime diagnostics {"slice": "nfcorpus_rank_choice", "p50 ms": "1131", "p95 ms": "6996", "p99 ms": "7522", "mean input tokens": "2380", "total input tokens": "709,228", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 298 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · nfcorpus_rank_graded · nDCG@10	36.73%	Setup & source JevBench Ranking slices: nfcorpus_rank_graded · nDCG@10 Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "nfcorpus_rank_graded", "queries": "298", "nDCG@10": "0.3673", "MRR@10": "0.6026", "Recall@10": "0.1720", "BM25 nDCG@10": "0.3319", "ceiling nDCG@10": "0.4651", "shortlist recall": "19.1%"}. Runtime diagnostics {"slice": "nfcorpus_rank_graded", "p50 ms": "582", "p95 ms": "1449", "p99 ms": "2023", "mean input tokens": "307", "total input tokens": "1,511,895", "retries": "1"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 298 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · nfcorpus_rank_graded · MRR@10	60.26%	Setup & source JevBench Ranking slices: nfcorpus_rank_graded · MRR@10 Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "nfcorpus_rank_graded", "queries": "298", "nDCG@10": "0.3673", "MRR@10": "0.6026", "Recall@10": "0.1720", "BM25 nDCG@10": "0.3319", "ceiling nDCG@10": "0.4651", "shortlist recall": "19.1%"}. Runtime diagnostics {"slice": "nfcorpus_rank_graded", "p50 ms": "582", "p95 ms": "1449", "p99 ms": "2023", "mean input tokens": "307", "total input tokens": "1,511,895", "retries": "1"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 298 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · nfcorpus_rank_graded · Recall@10	17.20%	Setup & source JevBench Ranking slices: nfcorpus_rank_graded · Recall@10 Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "nfcorpus_rank_graded", "queries": "298", "nDCG@10": "0.3673", "MRR@10": "0.6026", "Recall@10": "0.1720", "BM25 nDCG@10": "0.3319", "ceiling nDCG@10": "0.4651", "shortlist recall": "19.1%"}. Runtime diagnostics {"slice": "nfcorpus_rank_graded", "p50 ms": "582", "p95 ms": "1449", "p99 ms": "2023", "mean input tokens": "307", "total input tokens": "1,511,895", "retries": "1"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 298 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · nfcorpus_rank_noul · nDCG@10	37.46%	Setup & source JevBench Ranking slices: nfcorpus_rank_noul · nDCG@10 Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "nfcorpus_rank_noul", "queries": "298", "nDCG@10": "0.3746", "MRR@10": "0.6220", "Recall@10": "0.1723", "BM25 nDCG@10": "0.3319", "ceiling nDCG@10": "0.4651", "shortlist recall": "19.1%"}. Runtime diagnostics {"slice": "nfcorpus_rank_noul", "p50 ms": "2641", "p95 ms": "4668", "p99 ms": "5745", "mean input tokens": "2399", "total input tokens": "715,032", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 298 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · nfcorpus_rank_noul · MRR@10	62.20%	Setup & source JevBench Ranking slices: nfcorpus_rank_noul · MRR@10 Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "nfcorpus_rank_noul", "queries": "298", "nDCG@10": "0.3746", "MRR@10": "0.6220", "Recall@10": "0.1723", "BM25 nDCG@10": "0.3319", "ceiling nDCG@10": "0.4651", "shortlist recall": "19.1%"}. Runtime diagnostics {"slice": "nfcorpus_rank_noul", "p50 ms": "2641", "p95 ms": "4668", "p99 ms": "5745", "mean input tokens": "2399", "total input tokens": "715,032", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 298 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · nfcorpus_rank_noul · Recall@10	17.23%	Setup & source JevBench Ranking slices: nfcorpus_rank_noul · Recall@10 Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "nfcorpus_rank_noul", "queries": "298", "nDCG@10": "0.3746", "MRR@10": "0.6220", "Recall@10": "0.1723", "BM25 nDCG@10": "0.3319", "ceiling nDCG@10": "0.4651", "shortlist recall": "19.1%"}. Runtime diagnostics {"slice": "nfcorpus_rank_noul", "p50 ms": "2641", "p95 ms": "4668", "p99 ms": "5745", "mean input tokens": "2399", "total input tokens": "715,032", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 298 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · scifact_rank_choice · nDCG@10	72.55%	Setup & source JevBench Ranking slices: scifact_rank_choice · nDCG@10 Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "scifact_rank_choice", "queries": "300", "nDCG@10": "0.7255", "MRR@10": "0.7071", "Recall@10": "0.8027", "BM25 nDCG@10": "0.6605", "ceiling nDCG@10": "0.8300", "shortlist recall": "82.6%"}. Runtime diagnostics {"slice": "scifact_rank_choice", "p50 ms": "1042", "p95 ms": "2422", "p99 ms": "2853", "mean input tokens": "3003", "total input tokens": "901,036", "retries": "3"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 300 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · scifact_rank_choice · MRR@10	70.71%	Setup & source JevBench Ranking slices: scifact_rank_choice · MRR@10 Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "scifact_rank_choice", "queries": "300", "nDCG@10": "0.7255", "MRR@10": "0.7071", "Recall@10": "0.8027", "BM25 nDCG@10": "0.6605", "ceiling nDCG@10": "0.8300", "shortlist recall": "82.6%"}. Runtime diagnostics {"slice": "scifact_rank_choice", "p50 ms": "1042", "p95 ms": "2422", "p99 ms": "2853", "mean input tokens": "3003", "total input tokens": "901,036", "retries": "3"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 300 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · scifact_rank_choice · Recall@10	80.27%	Setup & source JevBench Ranking slices: scifact_rank_choice · Recall@10 Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "scifact_rank_choice", "queries": "300", "nDCG@10": "0.7255", "MRR@10": "0.7071", "Recall@10": "0.8027", "BM25 nDCG@10": "0.6605", "ceiling nDCG@10": "0.8300", "shortlist recall": "82.6%"}. Runtime diagnostics {"slice": "scifact_rank_choice", "p50 ms": "1042", "p95 ms": "2422", "p99 ms": "2853", "mean input tokens": "3003", "total input tokens": "901,036", "retries": "3"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 300 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · scifact_rank_noul · nDCG@10	74.33%	Setup & source JevBench Ranking slices: scifact_rank_noul · nDCG@10 Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "scifact_rank_noul", "queries": "300", "nDCG@10": "0.7433", "MRR@10": "0.7255", "Recall@10": "0.8226", "BM25 nDCG@10": "0.6605", "ceiling nDCG@10": "0.8300", "shortlist recall": "82.6%"}. Runtime diagnostics {"slice": "scifact_rank_noul", "p50 ms": "3568", "p95 ms": "7529", "p99 ms": "8694", "mean input tokens": "3020", "total input tokens": "905,961", "retries": "312"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 300 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · scifact_rank_noul · MRR@10	72.55%	Setup & source JevBench Ranking slices: scifact_rank_noul · MRR@10 Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "scifact_rank_noul", "queries": "300", "nDCG@10": "0.7433", "MRR@10": "0.7255", "Recall@10": "0.8226", "BM25 nDCG@10": "0.6605", "ceiling nDCG@10": "0.8300", "shortlist recall": "82.6%"}. Runtime diagnostics {"slice": "scifact_rank_noul", "p50 ms": "3568", "p95 ms": "7529", "p99 ms": "8694", "mean input tokens": "3020", "total input tokens": "905,961", "retries": "312"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 300 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · scifact_rank_noul · Recall@10	82.26%	Setup & source JevBench Ranking slices: scifact_rank_noul · Recall@10 Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "scifact_rank_noul", "queries": "300", "nDCG@10": "0.7433", "MRR@10": "0.7255", "Recall@10": "0.8226", "BM25 nDCG@10": "0.6605", "ceiling nDCG@10": "0.8300", "shortlist recall": "82.6%"}. Runtime diagnostics {"slice": "scifact_rank_noul", "p50 ms": "3568", "p95 ms": "7529", "p99 ms": "8694", "mean input tokens": "3020", "total input tokens": "905,961", "retries": "312"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 300 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · rewardbench2 · Factuality · accuracy	82.90%	Setup & source JevBench By subset: rewardbench2 · accuracy Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "rewardbench2", "subset": "Factuality", "n": "240", "accuracy": "82.9%"}. Runtime diagnostics {"slice": "rewardbench2", "p50 ms": "896", "p95 ms": "2057", "p99 ms": "2813", "mean input tokens": "1777", "total input tokens": "2,132,815", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 240 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · rewardbench2 · Focus · accuracy	95.80%	Setup & source JevBench By subset: rewardbench2 · accuracy Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "rewardbench2", "subset": "Focus", "n": "240", "accuracy": "95.8%"}. Runtime diagnostics {"slice": "rewardbench2", "p50 ms": "896", "p95 ms": "2057", "p99 ms": "2813", "mean input tokens": "1777", "total input tokens": "2,132,815", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 240 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · rewardbench2 · Math · accuracy	77.10%	Setup & source JevBench By subset: rewardbench2 · accuracy Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "rewardbench2", "subset": "Math", "n": "240", "accuracy": "77.1%"}. Runtime diagnostics {"slice": "rewardbench2", "p50 ms": "896", "p95 ms": "2057", "p99 ms": "2813", "mean input tokens": "1777", "total input tokens": "2,132,815", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 240 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · rewardbench2 · Precise IF · accuracy	47.90%	Setup & source JevBench By subset: rewardbench2 · accuracy Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "rewardbench2", "subset": "Precise IF", "n": "240", "accuracy": "47.9%"}. Runtime diagnostics {"slice": "rewardbench2", "p50 ms": "896", "p95 ms": "2057", "p99 ms": "2813", "mean input tokens": "1777", "total input tokens": "2,132,815", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 240 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · rewardbench2 · Safety · accuracy	92.90%	Setup & source JevBench By subset: rewardbench2 · accuracy Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "rewardbench2", "subset": "Safety", "n": "240", "accuracy": "92.9%"}. Runtime diagnostics {"slice": "rewardbench2", "p50 ms": "896", "p95 ms": "2057", "p99 ms": "2813", "mean input tokens": "1777", "total input tokens": "2,132,815", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 240 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · banking77 · original order	90.40%	Setup & source JevBench Option-order sensitivity: banking77 · original order Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "banking77", "families with both": "500", "original order": "90.4%", "permuted order": "89.4%", "both correct": "88.0%", "flipped by reordering": "3.8%"}. Runtime diagnostics {"slice": "banking77", "p50 ms": "854", "p95 ms": "1951", "p99 ms": "2792", "mean input tokens": "1599", "total input tokens": "1,599,482", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 500 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · banking77 · permuted order	89.40%	Setup & source JevBench Option-order sensitivity: banking77 · permuted order Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "banking77", "families with both": "500", "original order": "90.4%", "permuted order": "89.4%", "both correct": "88.0%", "flipped by reordering": "3.8%"}. Runtime diagnostics {"slice": "banking77", "p50 ms": "854", "p95 ms": "1951", "p99 ms": "2792", "mean input tokens": "1599", "total input tokens": "1,599,482", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 500 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · banking77 · both correct	88.00%	Setup & source JevBench Option-order sensitivity: banking77 · both correct Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "banking77", "families with both": "500", "original order": "90.4%", "permuted order": "89.4%", "both correct": "88.0%", "flipped by reordering": "3.8%"}. Runtime diagnostics {"slice": "banking77", "p50 ms": "854", "p95 ms": "1951", "p99 ms": "2792", "mean input tokens": "1599", "total input tokens": "1,599,482", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 500 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · banking77 · flipped by reordering	3.80%	Setup & source JevBench Option-order sensitivity: banking77 · flipped by reordering Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "banking77", "families with both": "500", "original order": "90.4%", "permuted order": "89.4%", "both correct": "88.0%", "flipped by reordering": "3.8%"}. Runtime diagnostics {"slice": "banking77", "p50 ms": "854", "p95 ms": "1951", "p99 ms": "2792", "mean input tokens": "1599", "total input tokens": "1,599,482", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 500 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · mcpbench_server_routing · original order	100.00%	Setup & source JevBench Option-order sensitivity: mcpbench_server_routing · original order Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "mcpbench_server_routing", "families with both": "56", "original order": "100.0%", "permuted order": "100.0%", "both correct": "100.0%", "flipped by reordering": "0.0%"}. Runtime diagnostics {"slice": "mcpbench_server_routing", "p50 ms": "852", "p95 ms": "1950", "p99 ms": "2237", "mean input tokens": "867", "total input tokens": "97,113", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 56 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · mcpbench_server_routing · permuted order	100.00%	Setup & source JevBench Option-order sensitivity: mcpbench_server_routing · permuted order Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "mcpbench_server_routing", "families with both": "56", "original order": "100.0%", "permuted order": "100.0%", "both correct": "100.0%", "flipped by reordering": "0.0%"}. Runtime diagnostics {"slice": "mcpbench_server_routing", "p50 ms": "852", "p95 ms": "1950", "p99 ms": "2237", "mean input tokens": "867", "total input tokens": "97,113", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 56 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · mcpbench_server_routing · both correct	100.00%	Setup & source JevBench Option-order sensitivity: mcpbench_server_routing · both correct Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "mcpbench_server_routing", "families with both": "56", "original order": "100.0%", "permuted order": "100.0%", "both correct": "100.0%", "flipped by reordering": "0.0%"}. Runtime diagnostics {"slice": "mcpbench_server_routing", "p50 ms": "852", "p95 ms": "1950", "p99 ms": "2237", "mean input tokens": "867", "total input tokens": "97,113", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 56 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · mcpbench_server_routing · flipped by reordering	0.00%	Setup & source JevBench Option-order sensitivity: mcpbench_server_routing · flipped by reordering Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "mcpbench_server_routing", "families with both": "56", "original order": "100.0%", "permuted order": "100.0%", "both correct": "100.0%", "flipped by reordering": "0.0%"}. Runtime diagnostics {"slice": "mcpbench_server_routing", "p50 ms": "852", "p95 ms": "1950", "p99 ms": "2237", "mean input tokens": "867", "total input tokens": "97,113", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 56 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · mcpbench_tool_selection · original order	100.00%	Setup & source JevBench Option-order sensitivity: mcpbench_tool_selection · original order Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "mcpbench_tool_selection", "families with both": "4", "original order": "100.0%", "permuted order": "100.0%", "both correct": "100.0%", "flipped by reordering": "0.0%"}. Runtime diagnostics {"slice": "mcpbench_tool_selection", "p50 ms": "773", "p95 ms": "2045", "p99 ms": "2180", "mean input tokens": "755", "total input tokens": "19,632", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 4 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · mcpbench_tool_selection · permuted order	100.00%	Setup & source JevBench Option-order sensitivity: mcpbench_tool_selection · permuted order Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "mcpbench_tool_selection", "families with both": "4", "original order": "100.0%", "permuted order": "100.0%", "both correct": "100.0%", "flipped by reordering": "0.0%"}. Runtime diagnostics {"slice": "mcpbench_tool_selection", "p50 ms": "773", "p95 ms": "2045", "p99 ms": "2180", "mean input tokens": "755", "total input tokens": "19,632", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 4 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · mcpbench_tool_selection · both correct	100.00%	Setup & source JevBench Option-order sensitivity: mcpbench_tool_selection · both correct Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "mcpbench_tool_selection", "families with both": "4", "original order": "100.0%", "permuted order": "100.0%", "both correct": "100.0%", "flipped by reordering": "0.0%"}. Runtime diagnostics {"slice": "mcpbench_tool_selection", "p50 ms": "773", "p95 ms": "2045", "p99 ms": "2180", "mean input tokens": "755", "total input tokens": "19,632", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 4 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · mcpbench_tool_selection · flipped by reordering	0.00%	Setup & source JevBench Option-order sensitivity: mcpbench_tool_selection · flipped by reordering Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "mcpbench_tool_selection", "families with both": "4", "original order": "100.0%", "permuted order": "100.0%", "both correct": "100.0%", "flipped by reordering": "0.0%"}. Runtime diagnostics {"slice": "mcpbench_tool_selection", "p50 ms": "773", "p95 ms": "2045", "p99 ms": "2180", "mean input tokens": "755", "total input tokens": "19,632", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 4 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · mcpmark_server_routing · original order	57.40%	Setup & source JevBench Option-order sensitivity: mcpmark_server_routing · original order Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "mcpmark_server_routing", "families with both": "169", "original order": "57.4%", "permuted order": "56.8%", "both correct": "55.0%", "flipped by reordering": "4.1%"}. Runtime diagnostics {"slice": "mcpmark_server_routing", "p50 ms": "545", "p95 ms": "1223", "p99 ms": "1453", "mean input tokens": "248", "total input tokens": "83,834", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 169 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · mcpmark_server_routing · permuted order	56.80%	Setup & source JevBench Option-order sensitivity: mcpmark_server_routing · permuted order Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "mcpmark_server_routing", "families with both": "169", "original order": "57.4%", "permuted order": "56.8%", "both correct": "55.0%", "flipped by reordering": "4.1%"}. Runtime diagnostics {"slice": "mcpmark_server_routing", "p50 ms": "545", "p95 ms": "1223", "p99 ms": "1453", "mean input tokens": "248", "total input tokens": "83,834", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 169 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · mcpmark_server_routing · both correct	55.00%	Setup & source JevBench Option-order sensitivity: mcpmark_server_routing · both correct Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "mcpmark_server_routing", "families with both": "169", "original order": "57.4%", "permuted order": "56.8%", "both correct": "55.0%", "flipped by reordering": "4.1%"}. Runtime diagnostics {"slice": "mcpmark_server_routing", "p50 ms": "545", "p95 ms": "1223", "p99 ms": "1453", "mean input tokens": "248", "total input tokens": "83,834", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 169 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · mcpmark_server_routing · flipped by reordering	4.10%	Setup & source JevBench Option-order sensitivity: mcpmark_server_routing · flipped by reordering Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "mcpmark_server_routing", "families with both": "169", "original order": "57.4%", "permuted order": "56.8%", "both correct": "55.0%", "flipped by reordering": "4.1%"}. Runtime diagnostics {"slice": "mcpmark_server_routing", "p50 ms": "545", "p95 ms": "1223", "p99 ms": "1453", "mean input tokens": "248", "total input tokens": "83,834", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 169 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · mcpmark_tool_selection · original order	100.00%	Setup & source JevBench Option-order sensitivity: mcpmark_tool_selection · original order Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "mcpmark_tool_selection", "families with both": "3", "original order": "100.0%", "permuted order": "100.0%", "both correct": "100.0%", "flipped by reordering": "0.0%"}. Runtime diagnostics {"slice": "mcpmark_tool_selection", "p50 ms": "683", "p95 ms": "3488", "p99 ms": "3488", "mean input tokens": "402", "total input tokens": "7,245", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 3 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · mcpmark_tool_selection · permuted order	100.00%	Setup & source JevBench Option-order sensitivity: mcpmark_tool_selection · permuted order Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "mcpmark_tool_selection", "families with both": "3", "original order": "100.0%", "permuted order": "100.0%", "both correct": "100.0%", "flipped by reordering": "0.0%"}. Runtime diagnostics {"slice": "mcpmark_tool_selection", "p50 ms": "683", "p95 ms": "3488", "p99 ms": "3488", "mean input tokens": "402", "total input tokens": "7,245", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 3 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · mcpmark_tool_selection · both correct	100.00%	Setup & source JevBench Option-order sensitivity: mcpmark_tool_selection · both correct Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "mcpmark_tool_selection", "families with both": "3", "original order": "100.0%", "permuted order": "100.0%", "both correct": "100.0%", "flipped by reordering": "0.0%"}. Runtime diagnostics {"slice": "mcpmark_tool_selection", "p50 ms": "683", "p95 ms": "3488", "p99 ms": "3488", "mean input tokens": "402", "total input tokens": "7,245", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 3 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · mcpmark_tool_selection · flipped by reordering	0.00%	Setup & source JevBench Option-order sensitivity: mcpmark_tool_selection · flipped by reordering Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "mcpmark_tool_selection", "families with both": "3", "original order": "100.0%", "permuted order": "100.0%", "both correct": "100.0%", "flipped by reordering": "0.0%"}. Runtime diagnostics {"slice": "mcpmark_tool_selection", "p50 ms": "683", "p95 ms": "3488", "p99 ms": "3488", "mean input tokens": "402", "total input tokens": "7,245", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 3 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · medmcqa · original order	72.80%	Setup & source JevBench Option-order sensitivity: medmcqa · original order Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "medmcqa", "families with both": "500", "original order": "72.8%", "permuted order": "72.4%", "both correct": "66.0%", "flipped by reordering": "13.2%"}. Runtime diagnostics {"slice": "medmcqa", "p50 ms": "590", "p95 ms": "1329", "p99 ms": "1806", "mean input tokens": "141", "total input tokens": "141,057", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 500 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · medmcqa · permuted order	72.40%	Setup & source JevBench Option-order sensitivity: medmcqa · permuted order Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "medmcqa", "families with both": "500", "original order": "72.8%", "permuted order": "72.4%", "both correct": "66.0%", "flipped by reordering": "13.2%"}. Runtime diagnostics {"slice": "medmcqa", "p50 ms": "590", "p95 ms": "1329", "p99 ms": "1806", "mean input tokens": "141", "total input tokens": "141,057", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 500 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · medmcqa · both correct	66.00%	Setup & source JevBench Option-order sensitivity: medmcqa · both correct Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "medmcqa", "families with both": "500", "original order": "72.8%", "permuted order": "72.4%", "both correct": "66.0%", "flipped by reordering": "13.2%"}. Runtime diagnostics {"slice": "medmcqa", "p50 ms": "590", "p95 ms": "1329", "p99 ms": "1806", "mean input tokens": "141", "total input tokens": "141,057", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 500 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · medmcqa · flipped by reordering	13.20%	Setup & source JevBench Option-order sensitivity: medmcqa · flipped by reordering Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "medmcqa", "families with both": "500", "original order": "72.8%", "permuted order": "72.4%", "both correct": "66.0%", "flipped by reordering": "13.2%"}. Runtime diagnostics {"slice": "medmcqa", "p50 ms": "590", "p95 ms": "1329", "p99 ms": "1806", "mean input tokens": "141", "total input tokens": "141,057", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 500 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · medqa_usmle · original order	84.40%	Setup & source JevBench Option-order sensitivity: medqa_usmle · original order Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "medqa_usmle", "families with both": "500", "original order": "84.4%", "permuted order": "85.4%", "both correct": "81.4%", "flipped by reordering": "7.0%"}. Runtime diagnostics {"slice": "medqa_usmle", "p50 ms": "596", "p95 ms": "1546", "p99 ms": "2754", "mean input tokens": "301", "total input tokens": "301,460", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 500 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · medqa_usmle · permuted order	85.40%	Setup & source JevBench Option-order sensitivity: medqa_usmle · permuted order Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "medqa_usmle", "families with both": "500", "original order": "84.4%", "permuted order": "85.4%", "both correct": "81.4%", "flipped by reordering": "7.0%"}. Runtime diagnostics {"slice": "medqa_usmle", "p50 ms": "596", "p95 ms": "1546", "p99 ms": "2754", "mean input tokens": "301", "total input tokens": "301,460", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 500 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · medqa_usmle · both correct	81.40%	Setup & source JevBench Option-order sensitivity: medqa_usmle · both correct Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "medqa_usmle", "families with both": "500", "original order": "84.4%", "permuted order": "85.4%", "both correct": "81.4%", "flipped by reordering": "7.0%"}. Runtime diagnostics {"slice": "medqa_usmle", "p50 ms": "596", "p95 ms": "1546", "p99 ms": "2754", "mean input tokens": "301", "total input tokens": "301,460", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 500 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · medqa_usmle · flipped by reordering	7.00%	Setup & source JevBench Option-order sensitivity: medqa_usmle · flipped by reordering Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "medqa_usmle", "families with both": "500", "original order": "84.4%", "permuted order": "85.4%", "both correct": "81.4%", "flipped by reordering": "7.0%"}. Runtime diagnostics {"slice": "medqa_usmle", "p50 ms": "596", "p95 ms": "1546", "p99 ms": "2754", "mean input tokens": "301", "total input tokens": "301,460", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 500 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · mmlu_pro · original order	66.20%	Setup & source JevBench Option-order sensitivity: mmlu_pro · original order Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "mmlu_pro", "families with both": "500", "original order": "66.2%", "permuted order": "62.8%", "both correct": "58.4%", "flipped by reordering": "12.2%"}. Runtime diagnostics {"slice": "mmlu_pro", "p50 ms": "625", "p95 ms": "1712", "p99 ms": "2715", "mean input tokens": "333", "total input tokens": "332,557", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 500 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · mmlu_pro · permuted order	62.80%	Setup & source JevBench Option-order sensitivity: mmlu_pro · permuted order Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "mmlu_pro", "families with both": "500", "original order": "66.2%", "permuted order": "62.8%", "both correct": "58.4%", "flipped by reordering": "12.2%"}. Runtime diagnostics {"slice": "mmlu_pro", "p50 ms": "625", "p95 ms": "1712", "p99 ms": "2715", "mean input tokens": "333", "total input tokens": "332,557", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 500 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · mmlu_pro · both correct	58.40%	Setup & source JevBench Option-order sensitivity: mmlu_pro · both correct Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "mmlu_pro", "families with both": "500", "original order": "66.2%", "permuted order": "62.8%", "both correct": "58.4%", "flipped by reordering": "12.2%"}. Runtime diagnostics {"slice": "mmlu_pro", "p50 ms": "625", "p95 ms": "1712", "p99 ms": "2715", "mean input tokens": "333", "total input tokens": "332,557", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 500 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · mmlu_pro · flipped by reordering	12.20%	Setup & source JevBench Option-order sensitivity: mmlu_pro · flipped by reordering Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "mmlu_pro", "families with both": "500", "original order": "66.2%", "permuted order": "62.8%", "both correct": "58.4%", "flipped by reordering": "12.2%"}. Runtime diagnostics {"slice": "mmlu_pro", "p50 ms": "625", "p95 ms": "1712", "p99 ms": "2715", "mean input tokens": "333", "total input tokens": "332,557", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 500 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · rewardbench2 · original order	79.30%	Setup & source JevBench Option-order sensitivity: rewardbench2 · original order Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "rewardbench2", "families with both": "594", "original order": "79.3%", "permuted order": "79.3%", "both correct": "73.4%", "flipped by reordering": "12.1%"}. Runtime diagnostics {"slice": "rewardbench2", "p50 ms": "896", "p95 ms": "2057", "p99 ms": "2813", "mean input tokens": "1777", "total input tokens": "2,132,815", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 594 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · rewardbench2 · permuted order	79.30%	Setup & source JevBench Option-order sensitivity: rewardbench2 · permuted order Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "rewardbench2", "families with both": "594", "original order": "79.3%", "permuted order": "79.3%", "both correct": "73.4%", "flipped by reordering": "12.1%"}. Runtime diagnostics {"slice": "rewardbench2", "p50 ms": "896", "p95 ms": "2057", "p99 ms": "2813", "mean input tokens": "1777", "total input tokens": "2,132,815", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 594 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · rewardbench2 · both correct	73.40%	Setup & source JevBench Option-order sensitivity: rewardbench2 · both correct Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "rewardbench2", "families with both": "594", "original order": "79.3%", "permuted order": "79.3%", "both correct": "73.4%", "flipped by reordering": "12.1%"}. Runtime diagnostics {"slice": "rewardbench2", "p50 ms": "896", "p95 ms": "2057", "p99 ms": "2813", "mean input tokens": "1777", "total input tokens": "2,132,815", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 594 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · rewardbench2 · flipped by reordering	12.10%	Setup & source JevBench Option-order sensitivity: rewardbench2 · flipped by reordering Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "rewardbench2", "families with both": "594", "original order": "79.3%", "permuted order": "79.3%", "both correct": "73.4%", "flipped by reordering": "12.1%"}. Runtime diagnostics {"slice": "rewardbench2", "p50 ms": "896", "p95 ms": "2057", "p99 ms": "2813", "mean input tokens": "1777", "total input tokens": "2,132,815", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 594 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · scienceqa_text · original order	94.40%	Setup & source JevBench Option-order sensitivity: scienceqa_text · original order Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "scienceqa_text", "families with both": "500", "original order": "94.4%", "permuted order": "94.0%", "both correct": "92.4%", "flipped by reordering": "3.6%"}. Runtime diagnostics {"slice": "scienceqa_text", "p50 ms": "896", "p95 ms": "2274", "p99 ms": "2831", "mean input tokens": "129", "total input tokens": "129,244", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 500 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · scienceqa_text · permuted order	94.00%	Setup & source JevBench Option-order sensitivity: scienceqa_text · permuted order Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "scienceqa_text", "families with both": "500", "original order": "94.4%", "permuted order": "94.0%", "both correct": "92.4%", "flipped by reordering": "3.6%"}. Runtime diagnostics {"slice": "scienceqa_text", "p50 ms": "896", "p95 ms": "2274", "p99 ms": "2831", "mean input tokens": "129", "total input tokens": "129,244", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 500 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · scienceqa_text · both correct	92.40%	Setup & source JevBench Option-order sensitivity: scienceqa_text · both correct Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "scienceqa_text", "families with both": "500", "original order": "94.4%", "permuted order": "94.0%", "both correct": "92.4%", "flipped by reordering": "3.6%"}. Runtime diagnostics {"slice": "scienceqa_text", "p50 ms": "896", "p95 ms": "2274", "p99 ms": "2831", "mean input tokens": "129", "total input tokens": "129,244", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 500 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench · scienceqa_text · flipped by reordering	3.60%	Setup & source JevBench Option-order sensitivity: scienceqa_text · flipped by reordering Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published row {"slice": "scienceqa_text", "families with both": "500", "original order": "94.4%", "permuted order": "94.0%", "both correct": "92.4%", "flipped by reordering": "3.6%"}. Runtime diagnostics {"slice": "scienceqa_text", "p50 ms": "896", "p95 ms": "2274", "p99 ms": "2831", "mean input tokens": "129", "total input tokens": "129,244", "retries": "0"}. Native complete report preserved; intervals group reordered twins across2000bootstrap resamples. Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11T20:01:00-0700 · Samples: 500 Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses Result source · Benchmark source
JevBench Sep26 · Index	76.60	Setup & source JevBench Sep26 · Index Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published category aggregate. Per-dataset normalized inputs: {"medmcqa": 72.6, "medqa_usmle": 84.9, "pubmedqa": 65.6, "mmlu_pro": 64.5, "scienceqa_text": 94.2, "mcpbench_server_routing": 100.0, "mcpbench_tool_selection": 100.0, "mcpmark_server_routing": 57.1, "mcpmark_tool_selection": 100.0, "banking77": 89.9, "sst5": 88.1, "aegis2": 84.0, "aegis2_response": 77.2, "atbench500": 68.8, "jailbreak_classification": 96.8, "prompt_injections": 81.0, "scifact_rank_noul": 74.3, "nfcorpus_rank_noul": 37.5, "helpsteer2_helpfulness": 55.5, "rewardbench2": 79.3} Publisher normalized per-dataset scores, averaged within six categories and then categories. Complete6/6; not Benchmark Heaven JevBench or an Examenos recomputation. Checked / reported: 2026-10-20 · Samples: Not published Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
JevBench Sep26 · Science	76.40	Setup & source JevBench Sep26 · Science Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published category aggregate. Per-dataset normalized inputs: {"medmcqa": 72.6, "medqa_usmle": 84.9, "pubmedqa": 65.6, "mmlu_pro": 64.5, "scienceqa_text": 94.2, "mcpbench_server_routing": 100.0, "mcpbench_tool_selection": 100.0, "mcpmark_server_routing": 57.1, "mcpmark_tool_selection": 100.0, "banking77": 89.9, "sst5": 88.1, "aegis2": 84.0, "aegis2_response": 77.2, "atbench500": 68.8, "jailbreak_classification": 96.8, "prompt_injections": 81.0, "scifact_rank_noul": 74.3, "nfcorpus_rank_noul": 37.5, "helpsteer2_helpfulness": 55.5, "rewardbench2": 79.3} Publisher normalized per-dataset scores, averaged within six categories and then categories. Complete6/6; not Benchmark Heaven JevBench or an Examenos recomputation. Checked / reported: 2026-10-20 · Samples: Not published Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
JevBench Sep26 · Tool calling	89.30	Setup & source JevBench Sep26 · Tool calling Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published category aggregate. Per-dataset normalized inputs: {"medmcqa": 72.6, "medqa_usmle": 84.9, "pubmedqa": 65.6, "mmlu_pro": 64.5, "scienceqa_text": 94.2, "mcpbench_server_routing": 100.0, "mcpbench_tool_selection": 100.0, "mcpmark_server_routing": 57.1, "mcpmark_tool_selection": 100.0, "banking77": 89.9, "sst5": 88.1, "aegis2": 84.0, "aegis2_response": 77.2, "atbench500": 68.8, "jailbreak_classification": 96.8, "prompt_injections": 81.0, "scifact_rank_noul": 74.3, "nfcorpus_rank_noul": 37.5, "helpsteer2_helpfulness": 55.5, "rewardbench2": 79.3} Publisher normalized per-dataset scores, averaged within six categories and then categories. Complete6/6; not Benchmark Heaven JevBench or an Examenos recomputation. Checked / reported: 2026-10-20 · Samples: Not published Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
JevBench Sep26 · Business	89.00	Setup & source JevBench Sep26 · Business Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published category aggregate. Per-dataset normalized inputs: {"medmcqa": 72.6, "medqa_usmle": 84.9, "pubmedqa": 65.6, "mmlu_pro": 64.5, "scienceqa_text": 94.2, "mcpbench_server_routing": 100.0, "mcpbench_tool_selection": 100.0, "mcpmark_server_routing": 57.1, "mcpmark_tool_selection": 100.0, "banking77": 89.9, "sst5": 88.1, "aegis2": 84.0, "aegis2_response": 77.2, "atbench500": 68.8, "jailbreak_classification": 96.8, "prompt_injections": 81.0, "scifact_rank_noul": 74.3, "nfcorpus_rank_noul": 37.5, "helpsteer2_helpfulness": 55.5, "rewardbench2": 79.3} Publisher normalized per-dataset scores, averaged within six categories and then categories. Complete6/6; not Benchmark Heaven JevBench or an Examenos recomputation. Checked / reported: 2026-10-20 · Samples: Not published Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
JevBench Sep26 · Selection	81.60	Setup & source JevBench Sep26 · Selection Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published category aggregate. Per-dataset normalized inputs: {"medmcqa": 72.6, "medqa_usmle": 84.9, "pubmedqa": 65.6, "mmlu_pro": 64.5, "scienceqa_text": 94.2, "mcpbench_server_routing": 100.0, "mcpbench_tool_selection": 100.0, "mcpmark_server_routing": 57.1, "mcpmark_tool_selection": 100.0, "banking77": 89.9, "sst5": 88.1, "aegis2": 84.0, "aegis2_response": 77.2, "atbench500": 68.8, "jailbreak_classification": 96.8, "prompt_injections": 81.0, "scifact_rank_noul": 74.3, "nfcorpus_rank_noul": 37.5, "helpsteer2_helpfulness": 55.5, "rewardbench2": 79.3} Publisher normalized per-dataset scores, averaged within six categories and then categories. Complete6/6; not Benchmark Heaven JevBench or an Examenos recomputation. Checked / reported: 2026-10-20 · Samples: Not published Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
JevBench Sep26 · Ranking	55.90	Setup & source JevBench Sep26 · Ranking Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published category aggregate. Per-dataset normalized inputs: {"medmcqa": 72.6, "medqa_usmle": 84.9, "pubmedqa": 65.6, "mmlu_pro": 64.5, "scienceqa_text": 94.2, "mcpbench_server_routing": 100.0, "mcpbench_tool_selection": 100.0, "mcpmark_server_routing": 57.1, "mcpmark_tool_selection": 100.0, "banking77": 89.9, "sst5": 88.1, "aegis2": 84.0, "aegis2_response": 77.2, "atbench500": 68.8, "jailbreak_classification": 96.8, "prompt_injections": 81.0, "scifact_rank_noul": 74.3, "nfcorpus_rank_noul": 37.5, "helpsteer2_helpfulness": 55.5, "rewardbench2": 79.3} Publisher normalized per-dataset scores, averaged within six categories and then categories. Complete6/6; not Benchmark Heaven JevBench or an Examenos recomputation. Checked / reported: 2026-10-20 · Samples: Not published Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
JevBench Sep26 · Judging	67.40	Setup & source JevBench Sep26 · Judging Scored 2026-10-11T20:01:00-0700. Model requested `or-inception-mercury-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request. Published category aggregate. Per-dataset normalized inputs: {"medmcqa": 72.6, "medqa_usmle": 84.9, "pubmedqa": 65.6, "mmlu_pro": 64.5, "scienceqa_text": 94.2, "mcpbench_server_routing": 100.0, "mcpbench_tool_selection": 100.0, "mcpmark_server_routing": 57.1, "mcpmark_tool_selection": 100.0, "banking77": 89.9, "sst5": 88.1, "aegis2": 84.0, "aegis2_response": 77.2, "atbench500": 68.8, "jailbreak_classification": 96.8, "prompt_injections": 81.0, "scifact_rank_noul": 74.3, "nfcorpus_rank_noul": 37.5, "helpsteer2_helpfulness": 55.5, "rewardbench2": 79.3} Publisher normalized per-dataset scores, averaged within six categories and then categories. Complete6/6; not Benchmark Heaven JevBench or an Examenos recomputation. Checked / reported: 2026-10-20 · Samples: Not published Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained Result source · Benchmark source

Independent · JevBench multilingual

Independent · Vals AI

Independent · ModelTested

Mercury Decide · Independent · ModelTested
Evaluation	Result	Evidence
ModelTested 56-case · All-case accuracy	93.80%	Setup & source Exact named hosted API observation; All-case accuracy. Exact OpenRouter inception/mercury-decide, runner1, Inception provider; per-case receipt dates11October2026. Executed weight SHA unknown. 56authored situations (19hard,14public,42hidden), each with2typed questions; score averages per-case accuracy. Confidence100×(1−mean multiclassBrier/2), not ECE. Rounded source cost0.0 is not a free tariff. 56 authored situations, including 19 hard cases, each with two typed questions; 14 public and 42 hidden cases. Source aggregate averages per-case decision accuracy. Confidence score is 100×(1−mean multiclass Brier/2), not ECE or a calibration probability. Small single-run benchmark; current HF access/size does not inherit this evaluator’s stale closed/API-only metadata. No tariff inferred from rounded cost 0.0. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11 · Samples: 56 Snapshot: 2026-10-18 · Revision: 081e8f37130a4a9077b8cffe62827d4c3b9e3b23 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
ModelTested 56-case · Hard-case accuracy	89.50%	Setup & source Exact named hosted API observation; Hard-case accuracy. Exact OpenRouter inception/mercury-decide, runner1, Inception provider; per-case receipt dates11October2026. Executed weight SHA unknown. 56authored situations (19hard,14public,42hidden), each with2typed questions; score averages per-case accuracy. Confidence100×(1−mean multiclassBrier/2), not ECE. Rounded source cost0.0 is not a free tariff. 56 authored situations, including 19 hard cases, each with two typed questions; 14 public and 42 hidden cases. Source aggregate averages per-case decision accuracy. Confidence score is 100×(1−mean multiclass Brier/2), not ECE or a calibration probability. Small single-run benchmark; current HF access/size does not inherit this evaluator’s stale closed/API-only metadata. No tariff inferred from rounded cost 0.0. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11 · Samples: 19 Snapshot: 2026-10-18 · Revision: 081e8f37130a4a9077b8cffe62827d4c3b9e3b23 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
ModelTested 56-case · Public-case accuracy	96.40%	Setup & source Exact named hosted API observation; Public-case accuracy. Exact OpenRouter inception/mercury-decide, runner1, Inception provider; per-case receipt dates11October2026. Executed weight SHA unknown. 56authored situations (19hard,14public,42hidden), each with2typed questions; score averages per-case accuracy. Confidence100×(1−mean multiclassBrier/2), not ECE. Rounded source cost0.0 is not a free tariff. 56 authored situations, including 19 hard cases, each with two typed questions; 14 public and 42 hidden cases. Source aggregate averages per-case decision accuracy. Confidence score is 100×(1−mean multiclass Brier/2), not ECE or a calibration probability. Small single-run benchmark; current HF access/size does not inherit this evaluator’s stale closed/API-only metadata. No tariff inferred from rounded cost 0.0. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11 · Samples: 14 Snapshot: 2026-10-18 · Revision: 081e8f37130a4a9077b8cffe62827d4c3b9e3b23 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
ModelTested 56-case · Hidden-case accuracy	92.90%	Setup & source Exact named hosted API observation; Hidden-case accuracy. Exact OpenRouter inception/mercury-decide, runner1, Inception provider; per-case receipt dates11October2026. Executed weight SHA unknown. 56authored situations (19hard,14public,42hidden), each with2typed questions; score averages per-case accuracy. Confidence100×(1−mean multiclassBrier/2), not ECE. Rounded source cost0.0 is not a free tariff. 56 authored situations, including 19 hard cases, each with two typed questions; 14 public and 42 hidden cases. Source aggregate averages per-case decision accuracy. Confidence score is 100×(1−mean multiclass Brier/2), not ECE or a calibration probability. Small single-run benchmark; current HF access/size does not inherit this evaluator’s stale closed/API-only metadata. No tariff inferred from rounded cost 0.0. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11 · Samples: 42 Snapshot: 2026-10-18 · Revision: 081e8f37130a4a9077b8cffe62827d4c3b9e3b23 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
ModelTested 56-case · Brier-based confidence score	95.10	Setup & source Exact named hosted API observation; Brier-based confidence score. Exact OpenRouter inception/mercury-decide, runner1, Inception provider; per-case receipt dates11October2026. Executed weight SHA unknown. 56authored situations (19hard,14public,42hidden), each with2typed questions; score averages per-case accuracy. Confidence100×(1−mean multiclassBrier/2), not ECE. Rounded source cost0.0 is not a free tariff. 56 authored situations, including 19 hard cases, each with two typed questions; 14 public and 42 hidden cases. Source aggregate averages per-case decision accuracy. Confidence score is 100×(1−mean multiclass Brier/2), not ECE or a calibration probability. Small single-run benchmark; current HF access/size does not inherit this evaluator’s stale closed/API-only metadata. No tariff inferred from rounded cost 0.0. Checked / reported: 2026-10-20 · Evaluated: 2026-10-11 · Samples: 56 Snapshot: 2026-10-18 · Revision: 081e8f37130a4a9077b8cffe62827d4c3b9e3b23 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source

Evaluator-published · Benchmark Heaven

Evaluator-published · etsabary development study

Mercury Decide · Independent · JevBench multilingual
Evaluation	Result	Evidence
JevBench multilingual · English baseline · overall · acc	81.56%	Setup & source JevBench multilingual · English baseline · overall · acc Mercury Decide, same named hosted route; English baseline; slice overall. {} Full source observations: {"n": 2016, "acc": 0.8156, "auroc": 0.9574, "slices": {"aegis2": {"n": 500, "acc": 0.84, "auroc": 0.9279, "yes": 0.464, "gold_yes": 0.5}, "aegis2_response": {"n": 500, "acc": 0.772, "auroc": 0.8876, "yes": 0.352, "gold_yes": 0.5}, "atbench500": {"n": 500, "acc": 0.688, "auroc": 0.9932, "yes": 0.188, "gold_yes": 0.5}, "jailbreak_classification": {"n": 400, "acc": 0.9675, "auroc": 0.993, "yes": 0.32, "gold_yes": 0.3525}, "prompt_injections": {"n": 116, "acc": 0.8103, "auroc": 0.9851, "yes": 0.3276, "gold_yes": 0.5172}}}. Translation provenance: {"zh": {"jailbreak_classification": {"cases": 400, "usable": 397}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 499}, "aegis2_response": {"cases": 500, "usable": 500}, "atbench500": {"cases": 500, "usable": 467}, "_segments": {"records": 5191, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "ar": {"jailbreak_classification": {"cases": 400, "usable": 399}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 499}, "atbench500": {"cases": 500, "usable": 493}, "_segments": {"records": 5089, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "hi": {"jailbreak_classification": {"cases": 400, "usable": 397}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 499}, "atbench500": {"cases": 500, "usable": 487}, "_segments": {"records": 14759, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "vi": {"jailbreak_classification": {"cases": 400, "usable": 394}, "prompt_injections": {"cases": 116, "usable": 115}, "aegis2": {"cases": 500, "usable": 497}, "aegis2_response": {"cases": 500, "usable": 499}, "atbench500": {"cases": 500, "usable": 463}, "_segments": {"records": 5145, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "sw": {"jailbreak_classification": {"cases": 400, "usable": 398}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 499}, "atbench500": {"cases": 500, "usable": 484}, "_segments": {"records": 5333, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "es": {"jailbreak_classification": {"cases": 400, "usable": 396}, "prompt_injections": {"cases": 116, "usable": 113}, "aegis2": {"cases": 500, "usable": 499}, "aegis2_response": {"cases": 500, "usable": 500}, "atbench500": {"cases": 500, "usable": 451}, "_segments": {"records": 6637, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "fr": {"jailbreak_classification": {"cases": 400, "usable": 395}, "prompt_injections": {"cases": 116, "usable": 115}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 500}, "atbench500": {"cases": 500, "usable": 459}, "_segments": {"records": 5132, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "ru": {"jailbreak_classification": {"cases": 400, "usable": 399}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 500}, "atbench500": {"cases": 500, "usable": 490}, "_segments": {"records": 5092, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}} Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation. Checked / reported: 2026-10-20 · Samples: 2016 Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
JevBench multilingual · English baseline · overall · auroc	0.9574 coefficient	Setup & source JevBench multilingual · English baseline · overall · auroc Mercury Decide, same named hosted route; English baseline; slice overall. {} Full source observations: {"n": 2016, "acc": 0.8156, "auroc": 0.9574, "slices": {"aegis2": {"n": 500, "acc": 0.84, "auroc": 0.9279, "yes": 0.464, "gold_yes": 0.5}, "aegis2_response": {"n": 500, "acc": 0.772, "auroc": 0.8876, "yes": 0.352, "gold_yes": 0.5}, "atbench500": {"n": 500, "acc": 0.688, "auroc": 0.9932, "yes": 0.188, "gold_yes": 0.5}, "jailbreak_classification": {"n": 400, "acc": 0.9675, "auroc": 0.993, "yes": 0.32, "gold_yes": 0.3525}, "prompt_injections": {"n": 116, "acc": 0.8103, "auroc": 0.9851, "yes": 0.3276, "gold_yes": 0.5172}}}. Translation provenance: {"zh": {"jailbreak_classification": {"cases": 400, "usable": 397}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 499}, "aegis2_response": {"cases": 500, "usable": 500}, "atbench500": {"cases": 500, "usable": 467}, "_segments": {"records": 5191, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "ar": {"jailbreak_classification": {"cases": 400, "usable": 399}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 499}, "atbench500": {"cases": 500, "usable": 493}, "_segments": {"records": 5089, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "hi": {"jailbreak_classification": {"cases": 400, "usable": 397}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 499}, "atbench500": {"cases": 500, "usable": 487}, "_segments": {"records": 14759, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "vi": {"jailbreak_classification": {"cases": 400, "usable": 394}, "prompt_injections": {"cases": 116, "usable": 115}, "aegis2": {"cases": 500, "usable": 497}, "aegis2_response": {"cases": 500, "usable": 499}, "atbench500": {"cases": 500, "usable": 463}, "_segments": {"records": 5145, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "sw": {"jailbreak_classification": {"cases": 400, "usable": 398}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 499}, "atbench500": {"cases": 500, "usable": 484}, "_segments": {"records": 5333, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "es": {"jailbreak_classification": {"cases": 400, "usable": 396}, "prompt_injections": {"cases": 116, "usable": 113}, "aegis2": {"cases": 500, "usable": 499}, "aegis2_response": {"cases": 500, "usable": 500}, "atbench500": {"cases": 500, "usable": 451}, "_segments": {"records": 6637, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "fr": {"jailbreak_classification": {"cases": 400, "usable": 395}, "prompt_injections": {"cases": 116, "usable": 115}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 500}, "atbench500": {"cases": 500, "usable": 459}, "_segments": {"records": 5132, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "ru": {"jailbreak_classification": {"cases": 400, "usable": 399}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 500}, "atbench500": {"cases": 500, "usable": 490}, "_segments": {"records": 5092, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}} Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation. Checked / reported: 2026-10-20 · Samples: 2016 Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
JevBench multilingual · English baseline · aegis2 · acc	84.00%	Setup & source JevBench multilingual · English baseline · aegis2 · acc Mercury Decide, same named hosted route; English baseline; slice aegis2. {} Full source observations: {"n": 500, "acc": 0.84, "auroc": 0.9279, "yes": 0.464, "gold_yes": 0.5}. Translation provenance: {"zh": {"jailbreak_classification": {"cases": 400, "usable": 397}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 499}, "aegis2_response": {"cases": 500, "usable": 500}, "atbench500": {"cases": 500, "usable": 467}, "_segments": {"records": 5191, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "ar": {"jailbreak_classification": {"cases": 400, "usable": 399}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 499}, "atbench500": {"cases": 500, "usable": 493}, "_segments": {"records": 5089, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "hi": {"jailbreak_classification": {"cases": 400, "usable": 397}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 499}, "atbench500": {"cases": 500, "usable": 487}, "_segments": {"records": 14759, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "vi": {"jailbreak_classification": {"cases": 400, "usable": 394}, "prompt_injections": {"cases": 116, "usable": 115}, "aegis2": {"cases": 500, "usable": 497}, "aegis2_response": {"cases": 500, "usable": 499}, "atbench500": {"cases": 500, "usable": 463}, "_segments": {"records": 5145, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "sw": {"jailbreak_classification": {"cases": 400, "usable": 398}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 499}, "atbench500": {"cases": 500, "usable": 484}, "_segments": {"records": 5333, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "es": {"jailbreak_classification": {"cases": 400, "usable": 396}, "prompt_injections": {"cases": 116, "usable": 113}, "aegis2": {"cases": 500, "usable": 499}, "aegis2_response": {"cases": 500, "usable": 500}, "atbench500": {"cases": 500, "usable": 451}, "_segments": {"records": 6637, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "fr": {"jailbreak_classification": {"cases": 400, "usable": 395}, "prompt_injections": {"cases": 116, "usable": 115}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 500}, "atbench500": {"cases": 500, "usable": 459}, "_segments": {"records": 5132, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "ru": {"jailbreak_classification": {"cases": 400, "usable": 399}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 500}, "atbench500": {"cases": 500, "usable": 490}, "_segments": {"records": 5092, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}} Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation. Checked / reported: 2026-10-20 · Samples: 500 Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
JevBench multilingual · English baseline · aegis2 · auroc	0.9279 coefficient	Setup & source JevBench multilingual · English baseline · aegis2 · auroc Mercury Decide, same named hosted route; English baseline; slice aegis2. {} Full source observations: {"n": 500, "acc": 0.84, "auroc": 0.9279, "yes": 0.464, "gold_yes": 0.5}. Translation provenance: {"zh": {"jailbreak_classification": {"cases": 400, "usable": 397}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 499}, "aegis2_response": {"cases": 500, "usable": 500}, "atbench500": {"cases": 500, "usable": 467}, "_segments": {"records": 5191, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "ar": {"jailbreak_classification": {"cases": 400, "usable": 399}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 499}, "atbench500": {"cases": 500, "usable": 493}, "_segments": {"records": 5089, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "hi": {"jailbreak_classification": {"cases": 400, "usable": 397}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 499}, "atbench500": {"cases": 500, "usable": 487}, "_segments": {"records": 14759, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "vi": {"jailbreak_classification": {"cases": 400, "usable": 394}, "prompt_injections": {"cases": 116, "usable": 115}, "aegis2": {"cases": 500, "usable": 497}, "aegis2_response": {"cases": 500, "usable": 499}, "atbench500": {"cases": 500, "usable": 463}, "_segments": {"records": 5145, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "sw": {"jailbreak_classification": {"cases": 400, "usable": 398}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 499}, "atbench500": {"cases": 500, "usable": 484}, "_segments": {"records": 5333, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "es": {"jailbreak_classification": {"cases": 400, "usable": 396}, "prompt_injections": {"cases": 116, "usable": 113}, "aegis2": {"cases": 500, "usable": 499}, "aegis2_response": {"cases": 500, "usable": 500}, "atbench500": {"cases": 500, "usable": 451}, "_segments": {"records": 6637, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "fr": {"jailbreak_classification": {"cases": 400, "usable": 395}, "prompt_injections": {"cases": 116, "usable": 115}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 500}, "atbench500": {"cases": 500, "usable": 459}, "_segments": {"records": 5132, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "ru": {"jailbreak_classification": {"cases": 400, "usable": 399}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 500}, "atbench500": {"cases": 500, "usable": 490}, "_segments": {"records": 5092, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}} Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation. Checked / reported: 2026-10-20 · Samples: 500 Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
JevBench multilingual · English baseline · aegis2 · yes	46.40%	Setup & source JevBench multilingual · English baseline · aegis2 · yes Mercury Decide, same named hosted route; English baseline; slice aegis2. {} Full source observations: {"n": 500, "acc": 0.84, "auroc": 0.9279, "yes": 0.464, "gold_yes": 0.5}. Translation provenance: {"zh": {"jailbreak_classification": {"cases": 400, "usable": 397}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 499}, "aegis2_response": {"cases": 500, "usable": 500}, "atbench500": {"cases": 500, "usable": 467}, "_segments": {"records": 5191, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "ar": {"jailbreak_classification": {"cases": 400, "usable": 399}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 499}, "atbench500": {"cases": 500, "usable": 493}, "_segments": {"records": 5089, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "hi": {"jailbreak_classification": {"cases": 400, "usable": 397}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 499}, "atbench500": {"cases": 500, "usable": 487}, "_segments": {"records": 14759, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "vi": {"jailbreak_classification": {"cases": 400, "usable": 394}, "prompt_injections": {"cases": 116, "usable": 115}, "aegis2": {"cases": 500, "usable": 497}, "aegis2_response": {"cases": 500, "usable": 499}, "atbench500": {"cases": 500, "usable": 463}, "_segments": {"records": 5145, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "sw": {"jailbreak_classification": {"cases": 400, "usable": 398}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 499}, "atbench500": {"cases": 500, "usable": 484}, "_segments": {"records": 5333, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "es": {"jailbreak_classification": {"cases": 400, "usable": 396}, "prompt_injections": {"cases": 116, "usable": 113}, "aegis2": {"cases": 500, "usable": 499}, "aegis2_response": {"cases": 500, "usable": 500}, "atbench500": {"cases": 500, "usable": 451}, "_segments": {"records": 6637, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "fr": {"jailbreak_classification": {"cases": 400, "usable": 395}, "prompt_injections": {"cases": 116, "usable": 115}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 500}, "atbench500": {"cases": 500, "usable": 459}, "_segments": {"records": 5132, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "ru": {"jailbreak_classification": {"cases": 400, "usable": 399}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 500}, "atbench500": {"cases": 500, "usable": 490}, "_segments": {"records": 5092, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}} Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation. Checked / reported: 2026-10-20 · Samples: 500 Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
JevBench multilingual · English baseline · aegis2_response · acc	77.20%	Setup & source JevBench multilingual · English baseline · aegis2_response · acc Mercury Decide, same named hosted route; English baseline; slice aegis2_response. {} Full source observations: {"n": 500, "acc": 0.772, "auroc": 0.8876, "yes": 0.352, "gold_yes": 0.5}. Translation provenance: {"zh": {"jailbreak_classification": {"cases": 400, "usable": 397}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 499}, "aegis2_response": {"cases": 500, "usable": 500}, "atbench500": {"cases": 500, "usable": 467}, "_segments": {"records": 5191, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "ar": {"jailbreak_classification": {"cases": 400, "usable": 399}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 499}, "atbench500": {"cases": 500, "usable": 493}, "_segments": {"records": 5089, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "hi": {"jailbreak_classification": {"cases": 400, "usable": 397}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 499}, "atbench500": {"cases": 500, "usable": 487}, "_segments": {"records": 14759, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "vi": {"jailbreak_classification": {"cases": 400, "usable": 394}, "prompt_injections": {"cases": 116, "usable": 115}, "aegis2": {"cases": 500, "usable": 497}, "aegis2_response": {"cases": 500, "usable": 499}, "atbench500": {"cases": 500, "usable": 463}, "_segments": {"records": 5145, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "sw": {"jailbreak_classification": {"cases": 400, "usable": 398}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 499}, "atbench500": {"cases": 500, "usable": 484}, "_segments": {"records": 5333, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "es": {"jailbreak_classification": {"cases": 400, "usable": 396}, "prompt_injections": {"cases": 116, "usable": 113}, "aegis2": {"cases": 500, "usable": 499}, "aegis2_response": {"cases": 500, "usable": 500}, "atbench500": {"cases": 500, "usable": 451}, "_segments": {"records": 6637, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "fr": {"jailbreak_classification": {"cases": 400, "usable": 395}, "prompt_injections": {"cases": 116, "usable": 115}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 500}, "atbench500": {"cases": 500, "usable": 459}, "_segments": {"records": 5132, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "ru": {"jailbreak_classification": {"cases": 400, "usable": 399}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 500}, "atbench500": {"cases": 500, "usable": 490}, "_segments": {"records": 5092, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}} Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation. Checked / reported: 2026-10-20 · Samples: 500 Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
JevBench multilingual · English baseline · aegis2_response · auroc	0.8876 coefficient	Setup & source JevBench multilingual · English baseline · aegis2_response · auroc Mercury Decide, same named hosted route; English baseline; slice aegis2_response. {} Full source observations: {"n": 500, "acc": 0.772, "auroc": 0.8876, "yes": 0.352, "gold_yes": 0.5}. Translation provenance: {"zh": {"jailbreak_classification": {"cases": 400, "usable": 397}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 499}, "aegis2_response": {"cases": 500, "usable": 500}, "atbench500": {"cases": 500, "usable": 467}, "_segments": {"records": 5191, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "ar": {"jailbreak_classification": {"cases": 400, "usable": 399}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 499}, "atbench500": {"cases": 500, "usable": 493}, "_segments": {"records": 5089, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "hi": {"jailbreak_classification": {"cases": 400, "usable": 397}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 499}, "atbench500": {"cases": 500, "usable": 487}, "_segments": {"records": 14759, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "vi": {"jailbreak_classification": {"cases": 400, "usable": 394}, "prompt_injections": {"cases": 116, "usable": 115}, "aegis2": {"cases": 500, "usable": 497}, "aegis2_response": {"cases": 500, "usable": 499}, "atbench500": {"cases": 500, "usable": 463}, "_segments": {"records": 5145, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "sw": {"jailbreak_classification": {"cases": 400, "usable": 398}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 499}, "atbench500": {"cases": 500, "usable": 484}, "_segments": {"records": 5333, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "es": {"jailbreak_classification": {"cases": 400, "usable": 396}, "prompt_injections": {"cases": 116, "usable": 113}, "aegis2": {"cases": 500, "usable": 499}, "aegis2_response": {"cases": 500, "usable": 500}, "atbench500": {"cases": 500, "usable": 451}, "_segments": {"records": 6637, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "fr": {"jailbreak_classification": {"cases": 400, "usable": 395}, "prompt_injections": {"cases": 116, "usable": 115}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 500}, "atbench500": {"cases": 500, "usable": 459}, "_segments": {"records": 5132, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "ru": {"jailbreak_classification": {"cases": 400, "usable": 399}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 500}, "atbench500": {"cases": 500, "usable": 490}, "_segments": {"records": 5092, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}} Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation. Checked / reported: 2026-10-20 · Samples: 500 Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
JevBench multilingual · English baseline · aegis2_response · yes	35.20%	Setup & source JevBench multilingual · English baseline · aegis2_response · yes Mercury Decide, same named hosted route; English baseline; slice aegis2_response. {} Full source observations: {"n": 500, "acc": 0.772, "auroc": 0.8876, "yes": 0.352, "gold_yes": 0.5}. Translation provenance: {"zh": {"jailbreak_classification": {"cases": 400, "usable": 397}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 499}, "aegis2_response": {"cases": 500, "usable": 500}, "atbench500": {"cases": 500, "usable": 467}, "_segments": {"records": 5191, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "ar": {"jailbreak_classification": {"cases": 400, "usable": 399}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 499}, "atbench500": {"cases": 500, "usable": 493}, "_segments": {"records": 5089, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "hi": {"jailbreak_classification": {"cases": 400, "usable": 397}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 499}, "atbench500": {"cases": 500, "usable": 487}, "_segments": {"records": 14759, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "vi": {"jailbreak_classification": {"cases": 400, "usable": 394}, "prompt_injections": {"cases": 116, "usable": 115}, "aegis2": {"cases": 500, "usable": 497}, "aegis2_response": {"cases": 500, "usable": 499}, "atbench500": {"cases": 500, "usable": 463}, "_segments": {"records": 5145, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "sw": {"jailbreak_classification": {"cases": 400, "usable": 398}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 499}, "atbench500": {"cases": 500, "usable": 484}, "_segments": {"records": 5333, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "es": {"jailbreak_classification": {"cases": 400, "usable": 396}, "prompt_injections": {"cases": 116, "usable": 113}, "aegis2": {"cases": 500, "usable": 499}, "aegis2_response": {"cases": 500, "usable": 500}, "atbench500": {"cases": 500, "usable": 451}, "_segments": {"records": 6637, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "fr": {"jailbreak_classification": {"cases": 400, "usable": 395}, "prompt_injections": {"cases": 116, "usable": 115}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 500}, "atbench500": {"cases": 500, "usable": 459}, "_segments": {"records": 5132, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "ru": {"jailbreak_classification": {"cases": 400, "usable": 399}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 500}, "atbench500": {"cases": 500, "usable": 490}, "_segments": {"records": 5092, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}} Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation. Checked / reported: 2026-10-20 · Samples: 500 Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
JevBench multilingual · English baseline · atbench500 · acc	68.80%	Setup & source JevBench multilingual · English baseline · atbench500 · acc Mercury Decide, same named hosted route; English baseline; slice atbench500. {} Full source observations: {"n": 500, "acc": 0.688, "auroc": 0.9932, "yes": 0.188, "gold_yes": 0.5}. Translation provenance: {"zh": {"jailbreak_classification": {"cases": 400, "usable": 397}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 499}, "aegis2_response": {"cases": 500, "usable": 500}, "atbench500": {"cases": 500, "usable": 467}, "_segments": {"records": 5191, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "ar": {"jailbreak_classification": {"cases": 400, "usable": 399}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 499}, "atbench500": {"cases": 500, "usable": 493}, "_segments": {"records": 5089, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "hi": {"jailbreak_classification": {"cases": 400, "usable": 397}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 499}, "atbench500": {"cases": 500, "usable": 487}, "_segments": {"records": 14759, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "vi": {"jailbreak_classification": {"cases": 400, "usable": 394}, "prompt_injections": {"cases": 116, "usable": 115}, "aegis2": {"cases": 500, "usable": 497}, "aegis2_response": {"cases": 500, "usable": 499}, "atbench500": {"cases": 500, "usable": 463}, "_segments": {"records": 5145, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "sw": {"jailbreak_classification": {"cases": 400, "usable": 398}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 499}, "atbench500": {"cases": 500, "usable": 484}, "_segments": {"records": 5333, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "es": {"jailbreak_classification": {"cases": 400, "usable": 396}, "prompt_injections": {"cases": 116, "usable": 113}, "aegis2": {"cases": 500, "usable": 499}, "aegis2_response": {"cases": 500, "usable": 500}, "atbench500": {"cases": 500, "usable": 451}, "_segments": {"records": 6637, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "fr": {"jailbreak_classification": {"cases": 400, "usable": 395}, "prompt_injections": {"cases": 116, "usable": 115}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 500}, "atbench500": {"cases": 500, "usable": 459}, "_segments": {"records": 5132, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "ru": {"jailbreak_classification": {"cases": 400, "usable": 399}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 500}, "atbench500": {"cases": 500, "usable": 490}, "_segments": {"records": 5092, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}} Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation. Checked / reported: 2026-10-20 · Samples: 500 Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
JevBench multilingual · English baseline · atbench500 · auroc	0.9932 coefficient	Setup & source JevBench multilingual · English baseline · atbench500 · auroc Mercury Decide, same named hosted route; English baseline; slice atbench500. {} Full source observations: {"n": 500, "acc": 0.688, "auroc": 0.9932, "yes": 0.188, "gold_yes": 0.5}. Translation provenance: {"zh": {"jailbreak_classification": {"cases": 400, "usable": 397}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 499}, "aegis2_response": {"cases": 500, "usable": 500}, "atbench500": {"cases": 500, "usable": 467}, "_segments": {"records": 5191, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "ar": {"jailbreak_classification": {"cases": 400, "usable": 399}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 499}, "atbench500": {"cases": 500, "usable": 493}, "_segments": {"records": 5089, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "hi": {"jailbreak_classification": {"cases": 400, "usable": 397}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 499}, "atbench500": {"cases": 500, "usable": 487}, "_segments": {"records": 14759, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "vi": {"jailbreak_classification": {"cases": 400, "usable": 394}, "prompt_injections": {"cases": 116, "usable": 115}, "aegis2": {"cases": 500, "usable": 497}, "aegis2_response": {"cases": 500, "usable": 499}, "atbench500": {"cases": 500, "usable": 463}, "_segments": {"records": 5145, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "sw": {"jailbreak_classification": {"cases": 400, "usable": 398}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 499}, "atbench500": {"cases": 500, "usable": 484}, "_segments": {"records": 5333, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "es": {"jailbreak_classification": {"cases": 400, "usable": 396}, "prompt_injections": {"cases": 116, "usable": 113}, "aegis2": {"cases": 500, "usable": 499}, "aegis2_response": {"cases": 500, "usable": 500}, "atbench500": {"cases": 500, "usable": 451}, "_segments": {"records": 6637, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "fr": {"jailbreak_classification": {"cases": 400, "usable": 395}, "prompt_injections": {"cases": 116, "usable": 115}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 500}, "atbench500": {"cases": 500, "usable": 459}, "_segments": {"records": 5132, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "ru": {"jailbreak_classification": {"cases": 400, "usable": 399}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 500}, "atbench500": {"cases": 500, "usable": 490}, "_segments": {"records": 5092, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}} Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation. Checked / reported: 2026-10-20 · Samples: 500 Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
JevBench multilingual · English baseline · atbench500 · yes	18.80%	Setup & source JevBench multilingual · English baseline · atbench500 · yes Mercury Decide, same named hosted route; English baseline; slice atbench500. {} Full source observations: {"n": 500, "acc": 0.688, "auroc": 0.9932, "yes": 0.188, "gold_yes": 0.5}. Translation provenance: {"zh": {"jailbreak_classification": {"cases": 400, "usable": 397}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 499}, "aegis2_response": {"cases": 500, "usable": 500}, "atbench500": {"cases": 500, "usable": 467}, "_segments": {"records": 5191, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "ar": {"jailbreak_classification": {"cases": 400, "usable": 399}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 499}, "atbench500": {"cases": 500, "usable": 493}, "_segments": {"records": 5089, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "hi": {"jailbreak_classification": {"cases": 400, "usable": 397}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 499}, "atbench500": {"cases": 500, "usable": 487}, "_segments": {"records": 14759, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "vi": {"jailbreak_classification": {"cases": 400, "usable": 394}, "prompt_injections": {"cases": 116, "usable": 115}, "aegis2": {"cases": 500, "usable": 497}, "aegis2_response": {"cases": 500, "usable": 499}, "atbench500": {"cases": 500, "usable": 463}, "_segments": {"records": 5145, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "sw": {"jailbreak_classification": {"cases": 400, "usable": 398}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 499}, "atbench500": {"cases": 500, "usable": 484}, "_segments": {"records": 5333, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "es": {"jailbreak_classification": {"cases": 400, "usable": 396}, "prompt_injections": {"cases": 116, "usable": 113}, "aegis2": {"cases": 500, "usable": 499}, "aegis2_response": {"cases": 500, "usable": 500}, "atbench500": {"cases": 500, "usable": 451}, "_segments": {"records": 6637, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "fr": {"jailbreak_classification": {"cases": 400, "usable": 395}, "prompt_injections": {"cases": 116, "usable": 115}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 500}, "atbench500": {"cases": 500, "usable": 459}, "_segments": {"records": 5132, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "ru": {"jailbreak_classification": {"cases": 400, "usable": 399}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 500}, "atbench500": {"cases": 500, "usable": 490}, "_segments": {"records": 5092, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}} Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation. Checked / reported: 2026-10-20 · Samples: 500 Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
JevBench multilingual · English baseline · jailbreak_classification · acc	96.75%	Setup & source JevBench multilingual · English baseline · jailbreak_classification · acc Mercury Decide, same named hosted route; English baseline; slice jailbreak_classification. {} Full source observations: {"n": 400, "acc": 0.9675, "auroc": 0.993, "yes": 0.32, "gold_yes": 0.3525}. Translation provenance: {"zh": {"jailbreak_classification": {"cases": 400, "usable": 397}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 499}, "aegis2_response": {"cases": 500, "usable": 500}, "atbench500": {"cases": 500, "usable": 467}, "_segments": {"records": 5191, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "ar": {"jailbreak_classification": {"cases": 400, "usable": 399}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 499}, "atbench500": {"cases": 500, "usable": 493}, "_segments": {"records": 5089, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "hi": {"jailbreak_classification": {"cases": 400, "usable": 397}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 499}, "atbench500": {"cases": 500, "usable": 487}, "_segments": {"records": 14759, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "vi": {"jailbreak_classification": {"cases": 400, "usable": 394}, "prompt_injections": {"cases": 116, "usable": 115}, "aegis2": {"cases": 500, "usable": 497}, "aegis2_response": {"cases": 500, "usable": 499}, "atbench500": {"cases": 500, "usable": 463}, "_segments": {"records": 5145, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "sw": {"jailbreak_classification": {"cases": 400, "usable": 398}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 499}, "atbench500": {"cases": 500, "usable": 484}, "_segments": {"records": 5333, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "es": {"jailbreak_classification": {"cases": 400, "usable": 396}, "prompt_injections": {"cases": 116, "usable": 113}, "aegis2": {"cases": 500, "usable": 499}, "aegis2_response": {"cases": 500, "usable": 500}, "atbench500": {"cases": 500, "usable": 451}, "_segments": {"records": 6637, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "fr": {"jailbreak_classification": {"cases": 400, "usable": 395}, "prompt_injections": {"cases": 116, "usable": 115}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 500}, "atbench500": {"cases": 500, "usable": 459}, "_segments": {"records": 5132, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "ru": {"jailbreak_classification": {"cases": 400, "usable": 399}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 500}, "atbench500": {"cases": 500, "usable": 490}, "_segments": {"records": 5092, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}} Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation. Checked / reported: 2026-10-20 · Samples: 400 Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
JevBench multilingual · English baseline · jailbreak_classification · auroc	0.9930 coefficient	Setup & source JevBench multilingual · English baseline · jailbreak_classification · auroc Mercury Decide, same named hosted route; English baseline; slice jailbreak_classification. {} Full source observations: {"n": 400, "acc": 0.9675, "auroc": 0.993, "yes": 0.32, "gold_yes": 0.3525}. Translation provenance: {"zh": {"jailbreak_classification": {"cases": 400, "usable": 397}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 499}, "aegis2_response": {"cases": 500, "usable": 500}, "atbench500": {"cases": 500, "usable": 467}, "_segments": {"records": 5191, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "ar": {"jailbreak_classification": {"cases": 400, "usable": 399}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 499}, "atbench500": {"cases": 500, "usable": 493}, "_segments": {"records": 5089, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "hi": {"jailbreak_classification": {"cases": 400, "usable": 397}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 499}, "atbench500": {"cases": 500, "usable": 487}, "_segments": {"records": 14759, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "vi": {"jailbreak_classification": {"cases": 400, "usable": 394}, "prompt_injections": {"cases": 116, "usable": 115}, "aegis2": {"cases": 500, "usable": 497}, "aegis2_response": {"cases": 500, "usable": 499}, "atbench500": {"cases": 500, "usable": 463}, "_segments": {"records": 5145, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "sw": {"jailbreak_classification": {"cases": 400, "usable": 398}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 499}, "atbench500": {"cases": 500, "usable": 484}, "_segments": {"records": 5333, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "es": {"jailbreak_classification": {"cases": 400, "usable": 396}, "prompt_injections": {"cases": 116, "usable": 113}, "aegis2": {"cases": 500, "usable": 499}, "aegis2_response": {"cases": 500, "usable": 500}, "atbench500": {"cases": 500, "usable": 451}, "_segments": {"records": 6637, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "fr": {"jailbreak_classification": {"cases": 400, "usable": 395}, "prompt_injections": {"cases": 116, "usable": 115}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 500}, "atbench500": {"cases": 500, "usable": 459}, "_segments": {"records": 5132, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "ru": {"jailbreak_classification": {"cases": 400, "usable": 399}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 500}, "atbench500": {"cases": 500, "usable": 490}, "_segments": {"records": 5092, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}} Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation. Checked / reported: 2026-10-20 · Samples: 400 Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
JevBench multilingual · English baseline · jailbreak_classification · yes	32.00%	Setup & source JevBench multilingual · English baseline · jailbreak_classification · yes Mercury Decide, same named hosted route; English baseline; slice jailbreak_classification. {} Full source observations: {"n": 400, "acc": 0.9675, "auroc": 0.993, "yes": 0.32, "gold_yes": 0.3525}. Translation provenance: {"zh": {"jailbreak_classification": {"cases": 400, "usable": 397}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 499}, "aegis2_response": {"cases": 500, "usable": 500}, "atbench500": {"cases": 500, "usable": 467}, "_segments": {"records": 5191, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "ar": {"jailbreak_classification": {"cases": 400, "usable": 399}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 499}, "atbench500": {"cases": 500, "usable": 493}, "_segments": {"records": 5089, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "hi": {"jailbreak_classification": {"cases": 400, "usable": 397}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 499}, "atbench500": {"cases": 500, "usable": 487}, "_segments": {"records": 14759, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "vi": {"jailbreak_classification": {"cases": 400, "usable": 394}, "prompt_injections": {"cases": 116, "usable": 115}, "aegis2": {"cases": 500, "usable": 497}, "aegis2_response": {"cases": 500, "usable": 499}, "atbench500": {"cases": 500, "usable": 463}, "_segments": {"records": 5145, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "sw": {"jailbreak_classification": {"cases": 400, "usable": 398}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 499}, "atbench500": {"cases": 500, "usable": 484}, "_segments": {"records": 5333, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "es": {"jailbreak_classification": {"cases": 400, "usable": 396}, "prompt_injections": {"cases": 116, "usable": 113}, "aegis2": {"cases": 500, "usable": 499}, "aegis2_response": {"cases": 500, "usable": 500}, "atbench500": {"cases": 500, "usable": 451}, "_segments": {"records": 6637, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "fr": {"jailbreak_classification": {"cases": 400, "usable": 395}, "prompt_injections": {"cases": 116, "usable": 115}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 500}, "atbench500": {"cases": 500, "usable": 459}, "_segments": {"records": 5132, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "ru": {"jailbreak_classification": {"cases": 400, "usable": 399}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 500}, "atbench500": {"cases": 500, "usable": 490}, "_segments": {"records": 5092, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}} Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation. Checked / reported: 2026-10-20 · Samples: 400 Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
JevBench multilingual · English baseline · prompt_injections · acc	81.03%	Setup & source JevBench multilingual · English baseline · prompt_injections · acc Mercury Decide, same named hosted route; English baseline; slice prompt_injections. {} Full source observations: {"n": 116, "acc": 0.8103, "auroc": 0.9851, "yes": 0.3276, "gold_yes": 0.5172}. Translation provenance: {"zh": {"jailbreak_classification": {"cases": 400, "usable": 397}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 499}, "aegis2_response": {"cases": 500, "usable": 500}, "atbench500": {"cases": 500, "usable": 467}, "_segments": {"records": 5191, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "ar": {"jailbreak_classification": {"cases": 400, "usable": 399}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 499}, "atbench500": {"cases": 500, "usable": 493}, "_segments": {"records": 5089, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "hi": {"jailbreak_classification": {"cases": 400, "usable": 397}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 499}, "atbench500": {"cases": 500, "usable": 487}, "_segments": {"records": 14759, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "vi": {"jailbreak_classification": {"cases": 400, "usable": 394}, "prompt_injections": {"cases": 116, "usable": 115}, "aegis2": {"cases": 500, "usable": 497}, "aegis2_response": {"cases": 500, "usable": 499}, "atbench500": {"cases": 500, "usable": 463}, "_segments": {"records": 5145, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "sw": {"jailbreak_classification": {"cases": 400, "usable": 398}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 499}, "atbench500": {"cases": 500, "usable": 484}, "_segments": {"records": 5333, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "es": {"jailbreak_classification": {"cases": 400, "usable": 396}, "prompt_injections": {"cases": 116, "usable": 113}, "aegis2": {"cases": 500, "usable": 499}, "aegis2_response": {"cases": 500, "usable": 500}, "atbench500": {"cases": 500, "usable": 451}, "_segments": {"records": 6637, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "fr": {"jailbreak_classification": {"cases": 400, "usable": 395}, "prompt_injections": {"cases": 116, "usable": 115}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 500}, "atbench500": {"cases": 500, "usable": 459}, "_segments": {"records": 5132, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "ru": {"jailbreak_classification": {"cases": 400, "usable": 399}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 500}, "atbench500": {"cases": 500, "usable": 490}, "_segments": {"records": 5092, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}} Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation. Checked / reported: 2026-10-20 · Samples: 116 Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
JevBench multilingual · English baseline · prompt_injections · auroc	0.9851 coefficient	Setup & source JevBench multilingual · English baseline · prompt_injections · auroc Mercury Decide, same named hosted route; English baseline; slice prompt_injections. {} Full source observations: {"n": 116, "acc": 0.8103, "auroc": 0.9851, "yes": 0.3276, "gold_yes": 0.5172}. Translation provenance: {"zh": {"jailbreak_classification": {"cases": 400, "usable": 397}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 499}, "aegis2_response": {"cases": 500, "usable": 500}, "atbench500": {"cases": 500, "usable": 467}, "_segments": {"records": 5191, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "ar": {"jailbreak_classification": {"cases": 400, "usable": 399}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 499}, "atbench500": {"cases": 500, "usable": 493}, "_segments": {"records": 5089, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "hi": {"jailbreak_classification": {"cases": 400, "usable": 397}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 499}, "atbench500": {"cases": 500, "usable": 487}, "_segments": {"records": 14759, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "vi": {"jailbreak_classification": {"cases": 400, "usable": 394}, "prompt_injections": {"cases": 116, "usable": 115}, "aegis2": {"cases": 500, "usable": 497}, "aegis2_response": {"cases": 500, "usable": 499}, "atbench500": {"cases": 500, "usable": 463}, "_segments": {"records": 5145, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "sw": {"jailbreak_classification": {"cases": 400, "usable": 398}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 499}, "atbench500": {"cases": 500, "usable": 484}, "_segments": {"records": 5333, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "es": {"jailbreak_classification": {"cases": 400, "usable": 396}, "prompt_injections": {"cases": 116, "usable": 113}, "aegis2": {"cases": 500, "usable": 499}, "aegis2_response": {"cases": 500, "usable": 500}, "atbench500": {"cases": 500, "usable": 451}, "_segments": {"records": 6637, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "fr": {"jailbreak_classification": {"cases": 400, "usable": 395}, "prompt_injections": {"cases": 116, "usable": 115}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 500}, "atbench500": {"cases": 500, "usable": 459}, "_segments": {"records": 5132, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "ru": {"jailbreak_classification": {"cases": 400, "usable": 399}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 500}, "atbench500": {"cases": 500, "usable": 490}, "_segments": {"records": 5092, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}} Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation. Checked / reported: 2026-10-20 · Samples: 116 Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
JevBench multilingual · English baseline · prompt_injections · yes	32.76%	Setup & source JevBench multilingual · English baseline · prompt_injections · yes Mercury Decide, same named hosted route; English baseline; slice prompt_injections. {} Full source observations: {"n": 116, "acc": 0.8103, "auroc": 0.9851, "yes": 0.3276, "gold_yes": 0.5172}. Translation provenance: {"zh": {"jailbreak_classification": {"cases": 400, "usable": 397}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 499}, "aegis2_response": {"cases": 500, "usable": 500}, "atbench500": {"cases": 500, "usable": 467}, "_segments": {"records": 5191, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "ar": {"jailbreak_classification": {"cases": 400, "usable": 399}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 499}, "atbench500": {"cases": 500, "usable": 493}, "_segments": {"records": 5089, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "hi": {"jailbreak_classification": {"cases": 400, "usable": 397}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 499}, "atbench500": {"cases": 500, "usable": 487}, "_segments": {"records": 14759, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "vi": {"jailbreak_classification": {"cases": 400, "usable": 394}, "prompt_injections": {"cases": 116, "usable": 115}, "aegis2": {"cases": 500, "usable": 497}, "aegis2_response": {"cases": 500, "usable": 499}, "atbench500": {"cases": 500, "usable": 463}, "_segments": {"records": 5145, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "sw": {"jailbreak_classification": {"cases": 400, "usable": 398}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 499}, "atbench500": {"cases": 500, "usable": 484}, "_segments": {"records": 5333, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "es": {"jailbreak_classification": {"cases": 400, "usable": 396}, "prompt_injections": {"cases": 116, "usable": 113}, "aegis2": {"cases": 500, "usable": 499}, "aegis2_response": {"cases": 500, "usable": 500}, "atbench500": {"cases": 500, "usable": 451}, "_segments": {"records": 6637, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "fr": {"jailbreak_classification": {"cases": 400, "usable": 395}, "prompt_injections": {"cases": 116, "usable": 115}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 500}, "atbench500": {"cases": 500, "usable": 459}, "_segments": {"records": 5132, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "ru": {"jailbreak_classification": {"cases": 400, "usable": 399}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 500}, "atbench500": {"cases": 500, "usable": 490}, "_segments": {"records": 5092, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}} Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation. Checked / reported: 2026-10-20 · Samples: 116 Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
JevBench multilingual · Chinese · overall · acc	81.26%	Setup & source JevBench multilingual · Chinese · overall · acc Mercury Decide, same named hosted route; Chinese; slice overall. {"answered": 1979, "cases": 1979, "complete": true} Full source observations: {"n": 1979, "acc": 0.8126, "en_acc": 0.8158, "delta": -0.0033, "ci": [-0.0179, 0.0116], "significant": false, "auroc": 0.9508, "en_auroc": 0.9572, "slices": {"aegis2": {"n": 499, "acc": 0.8437, "en_acc": 0.8397, "auroc": 0.9205, "en_auroc": 0.9276, "yes": 0.4729, "en_yes": 0.4649, "gold_yes": 0.501}, "aegis2_response": {"n": 500, "acc": 0.758, "en_acc": 0.772, "auroc": 0.8726, "en_auroc": 0.8876, "yes": 0.37, "en_yes": 0.352, "gold_yes": 0.5}, "atbench500": {"n": 467, "acc": 0.6702, "en_acc": 0.6874, "auroc": 0.9865, "en_auroc": 0.9924, "yes": 0.1627, "en_yes": 0.1799, "gold_yes": 0.4925}, "jailbreak_classification": {"n": 397, "acc": 0.9547, "en_acc": 0.9698, "auroc": 0.9929, "en_auroc": 0.9933, "yes": 0.3174, "en_yes": 0.3174, "gold_yes": 0.3476}, "prompt_injections": {"n": 116, "acc": 0.8362, "en_acc": 0.8103, "auroc": 0.9815, "en_auroc": 0.9851, "yes": 0.3534, "en_yes": 0.3276, "gold_yes": 0.5172}}}. Translation provenance: {"zh": {"jailbreak_classification": {"cases": 400, "usable": 397}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 499}, "aegis2_response": {"cases": 500, "usable": 500}, "atbench500": {"cases": 500, "usable": 467}, "_segments": {"records": 5191, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "ar": {"jailbreak_classification": {"cases": 400, "usable": 399}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 499}, "atbench500": {"cases": 500, "usable": 493}, "_segments": {"records": 5089, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "hi": {"jailbreak_classification": {"cases": 400, "usable": 397}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 499}, "atbench500": {"cases": 500, "usable": 487}, "_segments": {"records": 14759, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "vi": {"jailbreak_classification": {"cases": 400, "usable": 394}, "prompt_injections": {"cases": 116, "usable": 115}, "aegis2": {"cases": 500, "usable": 497}, "aegis2_response": {"cases": 500, "usable": 499}, "atbench500": {"cases": 500, "usable": 463}, "_segments": {"records": 5145, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "sw": {"jailbreak_classification": {"cases": 400, "usable": 398}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 499}, "atbench500": {"cases": 500, "usable": 484}, "_segments": {"records": 5333, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "es": {"jailbreak_classification": {"cases": 400, "usable": 396}, "prompt_injections": {"cases": 116, "usable": 113}, "aegis2": {"cases": 500, "usable": 499}, "aegis2_response": {"cases": 500, "usable": 500}, "atbench500": {"cases": 500, "usable": 451}, "_segments": {"records": 6637, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "fr": {"jailbreak_classification": {"cases": 400, "usable": 395}, "prompt_injections": {"cases": 116, "usable": 115}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 500}, "atbench500": {"cases": 500, "usable": 459}, "_segments": {"records": 5132, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "ru": {"jailbreak_classification": {"cases": 400, "usable": 399}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 500}, "atbench500": {"cases": 500, "usable": 490}, "_segments": {"records": 5092, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}} Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation. Checked / reported: 2026-10-20 · Samples: 1979 Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
JevBench multilingual · Chinese · overall · auroc	0.9508 coefficient	Setup & source JevBench multilingual · Chinese · overall · auroc Mercury Decide, same named hosted route; Chinese; slice overall. {"answered": 1979, "cases": 1979, "complete": true} Full source observations: {"n": 1979, "acc": 0.8126, "en_acc": 0.8158, "delta": -0.0033, "ci": [-0.0179, 0.0116], "significant": false, "auroc": 0.9508, "en_auroc": 0.9572, "slices": {"aegis2": {"n": 499, "acc": 0.8437, "en_acc": 0.8397, "auroc": 0.9205, "en_auroc": 0.9276, "yes": 0.4729, "en_yes": 0.4649, "gold_yes": 0.501}, "aegis2_response": {"n": 500, "acc": 0.758, "en_acc": 0.772, "auroc": 0.8726, "en_auroc": 0.8876, "yes": 0.37, "en_yes": 0.352, "gold_yes": 0.5}, "atbench500": {"n": 467, "acc": 0.6702, "en_acc": 0.6874, "auroc": 0.9865, "en_auroc": 0.9924, "yes": 0.1627, "en_yes": 0.1799, "gold_yes": 0.4925}, "jailbreak_classification": {"n": 397, "acc": 0.9547, "en_acc": 0.9698, "auroc": 0.9929, "en_auroc": 0.9933, "yes": 0.3174, "en_yes": 0.3174, "gold_yes": 0.3476}, "prompt_injections": {"n": 116, "acc": 0.8362, "en_acc": 0.8103, "auroc": 0.9815, "en_auroc": 0.9851, "yes": 0.3534, "en_yes": 0.3276, "gold_yes": 0.5172}}}. Translation provenance: {"zh": {"jailbreak_classification": {"cases": 400, "usable": 397}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 499}, "aegis2_response": {"cases": 500, "usable": 500}, "atbench500": {"cases": 500, "usable": 467}, "_segments": {"records": 5191, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "ar": {"jailbreak_classification": {"cases": 400, "usable": 399}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 499}, "atbench500": {"cases": 500, "usable": 493}, "_segments": {"records": 5089, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "hi": {"jailbreak_classification": {"cases": 400, "usable": 397}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 499}, "atbench500": {"cases": 500, "usable": 487}, "_segments": {"records": 14759, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "vi": {"jailbreak_classification": {"cases": 400, "usable": 394}, "prompt_injections": {"cases": 116, "usable": 115}, "aegis2": {"cases": 500, "usable": 497}, "aegis2_response": {"cases": 500, "usable": 499}, "atbench500": {"cases": 500, "usable": 463}, "_segments": {"records": 5145, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "sw": {"jailbreak_classification": {"cases": 400, "usable": 398}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 499}, "atbench500": {"cases": 500, "usable": 484}, "_segments": {"records": 5333, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "es": {"jailbreak_classification": {"cases": 400, "usable": 396}, "prompt_injections": {"cases": 116, "usable": 113}, "aegis2": {"cases": 500, "usable": 499}, "aegis2_response": {"cases": 500, "usable": 500}, "atbench500": {"cases": 500, "usable": 451}, "_segments": {"records": 6637, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "fr": {"jailbreak_classification": {"cases": 400, "usable": 395}, "prompt_injections": {"cases": 116, "usable": 115}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 500}, "atbench500": {"cases": 500, "usable": 459}, "_segments": {"records": 5132, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "ru": {"jailbreak_classification": {"cases": 400, "usable": 399}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 500}, "atbench500": {"cases": 500, "usable": 490}, "_segments": {"records": 5092, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}} Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation. Checked / reported: 2026-10-20 · Samples: 1979 Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
JevBench multilingual · Chinese · overall · en_acc	81.58%	Setup & source JevBench multilingual · Chinese · overall · en_acc Mercury Decide, same named hosted route; Chinese; slice overall. {"answered": 1979, "cases": 1979, "complete": true} Full source observations: {"n": 1979, "acc": 0.8126, "en_acc": 0.8158, "delta": -0.0033, "ci": [-0.0179, 0.0116], "significant": false, "auroc": 0.9508, "en_auroc": 0.9572, "slices": {"aegis2": {"n": 499, "acc": 0.8437, "en_acc": 0.8397, "auroc": 0.9205, "en_auroc": 0.9276, "yes": 0.4729, "en_yes": 0.4649, "gold_yes": 0.501}, "aegis2_response": {"n": 500, "acc": 0.758, "en_acc": 0.772, "auroc": 0.8726, "en_auroc": 0.8876, "yes": 0.37, "en_yes": 0.352, "gold_yes": 0.5}, "atbench500": {"n": 467, "acc": 0.6702, "en_acc": 0.6874, "auroc": 0.9865, "en_auroc": 0.9924, "yes": 0.1627, "en_yes": 0.1799, "gold_yes": 0.4925}, "jailbreak_classification": {"n": 397, "acc": 0.9547, "en_acc": 0.9698, "auroc": 0.9929, "en_auroc": 0.9933, "yes": 0.3174, "en_yes": 0.3174, "gold_yes": 0.3476}, "prompt_injections": {"n": 116, "acc": 0.8362, "en_acc": 0.8103, "auroc": 0.9815, "en_auroc": 0.9851, "yes": 0.3534, "en_yes": 0.3276, "gold_yes": 0.5172}}}. Translation provenance: {"zh": {"jailbreak_classification": {"cases": 400, "usable": 397}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 499}, "aegis2_response": {"cases": 500, "usable": 500}, "atbench500": {"cases": 500, "usable": 467}, "_segments": {"records": 5191, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "ar": {"jailbreak_classification": {"cases": 400, "usable": 399}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 499}, "atbench500": {"cases": 500, "usable": 493}, "_segments": {"records": 5089, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "hi": {"jailbreak_classification": {"cases": 400, "usable": 397}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 499}, "atbench500": {"cases": 500, "usable": 487}, "_segments": {"records": 14759, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "vi": {"jailbreak_classification": {"cases": 400, "usable": 394}, "prompt_injections": {"cases": 116, "usable": 115}, "aegis2": {"cases": 500, "usable": 497}, "aegis2_response": {"cases": 500, "usable": 499}, "atbench500": {"cases": 500, "usable": 463}, "_segments": {"records": 5145, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "sw": {"jailbreak_classification": {"cases": 400, "usable": 398}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 499}, "atbench500": {"cases": 500, "usable": 484}, "_segments": {"records": 5333, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "es": {"jailbreak_classification": {"cases": 400, "usable": 396}, "prompt_injections": {"cases": 116, "usable": 113}, "aegis2": {"cases": 500, "usable": 499}, "aegis2_response": {"cases": 500, "usable": 500}, "atbench500": {"cases": 500, "usable": 451}, "_segments": {"records": 6637, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "fr": {"jailbreak_classification": {"cases": 400, "usable": 395}, "prompt_injections": {"cases": 116, "usable": 115}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 500}, "atbench500": {"cases": 500, "usable": 459}, "_segments": {"records": 5132, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "ru": {"jailbreak_classification": {"cases": 400, "usable": 399}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 500}, "atbench500": {"cases": 500, "usable": 490}, "_segments": {"records": 5092, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}} Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation. Checked / reported: 2026-10-20 · Samples: 1979 Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
JevBench multilingual · Chinese · overall · en_auroc	0.9572 coefficient	Setup & source JevBench multilingual · Chinese · overall · en_auroc Mercury Decide, same named hosted route; Chinese; slice overall. {"answered": 1979, "cases": 1979, "complete": true} Full source observations: {"n": 1979, "acc": 0.8126, "en_acc": 0.8158, "delta": -0.0033, "ci": [-0.0179, 0.0116], "significant": false, "auroc": 0.9508, "en_auroc": 0.9572, "slices": {"aegis2": {"n": 499, "acc": 0.8437, "en_acc": 0.8397, "auroc": 0.9205, "en_auroc": 0.9276, "yes": 0.4729, "en_yes": 0.4649, "gold_yes": 0.501}, "aegis2_response": {"n": 500, "acc": 0.758, "en_acc": 0.772, "auroc": 0.8726, "en_auroc": 0.8876, "yes": 0.37, "en_yes": 0.352, "gold_yes": 0.5}, "atbench500": {"n": 467, "acc": 0.6702, "en_acc": 0.6874, "auroc": 0.9865, "en_auroc": 0.9924, "yes": 0.1627, "en_yes": 0.1799, "gold_yes": 0.4925}, "jailbreak_classification": {"n": 397, "acc": 0.9547, "en_acc": 0.9698, "auroc": 0.9929, "en_auroc": 0.9933, "yes": 0.3174, "en_yes": 0.3174, "gold_yes": 0.3476}, "prompt_injections": {"n": 116, "acc": 0.8362, "en_acc": 0.8103, "auroc": 0.9815, "en_auroc": 0.9851, "yes": 0.3534, "en_yes": 0.3276, "gold_yes": 0.5172}}}. Translation provenance: {"zh": {"jailbreak_classification": {"cases": 400, "usable": 397}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 499}, "aegis2_response": {"cases": 500, "usable": 500}, "atbench500": {"cases": 500, "usable": 467}, "_segments": {"records": 5191, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "ar": {"jailbreak_classification": {"cases": 400, "usable": 399}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 499}, "atbench500": {"cases": 500, "usable": 493}, "_segments": {"records": 5089, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "hi": {"jailbreak_classification": {"cases": 400, "usable": 397}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 499}, "atbench500": {"cases": 500, "usable": 487}, "_segments": {"records": 14759, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "vi": {"jailbreak_classification": {"cases": 400, "usable": 394}, "prompt_injections": {"cases": 116, "usable": 115}, "aegis2": {"cases": 500, "usable": 497}, "aegis2_response": {"cases": 500, "usable": 499}, "atbench500": {"cases": 500, "usable": 463}, "_segments": {"records": 5145, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "sw": {"jailbreak_classification": {"cases": 400, "usable": 398}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 499}, "atbench500": {"cases": 500, "usable": 484}, "_segments": {"records": 5333, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "es": {"jailbreak_classification": {"cases": 400, "usable": 396}, "prompt_injections": {"cases": 116, "usable": 113}, "aegis2": {"cases": 500, "usable": 499}, "aegis2_response": {"cases": 500, "usable": 500}, "atbench500": {"cases": 500, "usable": 451}, "_segments": {"records": 6637, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "fr": {"jailbreak_classification": {"cases": 400, "usable": 395}, "prompt_injections": {"cases": 116, "usable": 115}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 500}, "atbench500": {"cases": 500, "usable": 459}, "_segments": {"records": 5132, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "ru": {"jailbreak_classification": {"cases": 400, "usable": 399}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 500}, "atbench500": {"cases": 500, "usable": 490}, "_segments": {"records": 5092, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}} Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation. Checked / reported: 2026-10-20 · Samples: 1979 Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
JevBench multilingual · Chinese · overall · delta	-0.33 pp	Setup & source JevBench multilingual · Chinese · overall · delta Mercury Decide, same named hosted route; Chinese; slice overall. {"answered": 1979, "cases": 1979, "complete": true} Full source observations: {"n": 1979, "acc": 0.8126, "en_acc": 0.8158, "delta": -0.0033, "ci": [-0.0179, 0.0116], "significant": false, "auroc": 0.9508, "en_auroc": 0.9572, "slices": {"aegis2": {"n": 499, "acc": 0.8437, "en_acc": 0.8397, "auroc": 0.9205, "en_auroc": 0.9276, "yes": 0.4729, "en_yes": 0.4649, "gold_yes": 0.501}, "aegis2_response": {"n": 500, "acc": 0.758, "en_acc": 0.772, "auroc": 0.8726, "en_auroc": 0.8876, "yes": 0.37, "en_yes": 0.352, "gold_yes": 0.5}, "atbench500": {"n": 467, "acc": 0.6702, "en_acc": 0.6874, "auroc": 0.9865, "en_auroc": 0.9924, "yes": 0.1627, "en_yes": 0.1799, "gold_yes": 0.4925}, "jailbreak_classification": {"n": 397, "acc": 0.9547, "en_acc": 0.9698, "auroc": 0.9929, "en_auroc": 0.9933, "yes": 0.3174, "en_yes": 0.3174, "gold_yes": 0.3476}, "prompt_injections": {"n": 116, "acc": 0.8362, "en_acc": 0.8103, "auroc": 0.9815, "en_auroc": 0.9851, "yes": 0.3534, "en_yes": 0.3276, "gold_yes": 0.5172}}}. Translation provenance: {"zh": {"jailbreak_classification": {"cases": 400, "usable": 397}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 499}, "aegis2_response": {"cases": 500, "usable": 500}, "atbench500": {"cases": 500, "usable": 467}, "_segments": {"records": 5191, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "ar": {"jailbreak_classification": {"cases": 400, "usable": 399}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 499}, "atbench500": {"cases": 500, "usable": 493}, "_segments": {"records": 5089, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "hi": {"jailbreak_classification": {"cases": 400, "usable": 397}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 499}, "atbench500": {"cases": 500, "usable": 487}, "_segments": {"records": 14759, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "vi": {"jailbreak_classification": {"cases": 400, "usable": 394}, "prompt_injections": {"cases": 116, "usable": 115}, "aegis2": {"cases": 500, "usable": 497}, "aegis2_response": {"cases": 500, "usable": 499}, "atbench500": {"cases": 500, "usable": 463}, "_segments": {"records": 5145, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "sw": {"jailbreak_classification": {"cases": 400, "usable": 398}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 499}, "atbench500": {"cases": 500, "usable": 484}, "_segments": {"records": 5333, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "es": {"jailbreak_classification": {"cases": 400, "usable": 396}, "prompt_injections": {"cases": 116, "usable": 113}, "aegis2": {"cases": 500, "usable": 499}, "aegis2_response": {"cases": 500, "usable": 500}, "atbench500": {"cases": 500, "usable": 451}, "_segments": {"records": 6637, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "fr": {"jailbreak_classification": {"cases": 400, "usable": 395}, "prompt_injections": {"cases": 116, "usable": 115}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 500}, "atbench500": {"cases": 500, "usable": 459}, "_segments": {"records": 5132, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "ru": {"jailbreak_classification": {"cases": 400, "usable": 399}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 500}, "atbench500": {"cases": 500, "usable": 490}, "_segments": {"records": 5092, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}} Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation. Checked / reported: 2026-10-20 · Samples: 1979 Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
JevBench multilingual · Chinese · aegis2 · acc	84.37%	Setup & source JevBench multilingual · Chinese · aegis2 · acc Mercury Decide, same named hosted route; Chinese; slice aegis2. {"answered": 1979, "cases": 1979, "complete": true} Full source observations: {"n": 499, "acc": 0.8437, "en_acc": 0.8397, "auroc": 0.9205, "en_auroc": 0.9276, "yes": 0.4729, "en_yes": 0.4649, "gold_yes": 0.501}. Translation provenance: {"zh": {"jailbreak_classification": {"cases": 400, "usable": 397}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 499}, "aegis2_response": {"cases": 500, "usable": 500}, "atbench500": {"cases": 500, "usable": 467}, "_segments": {"records": 5191, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "ar": {"jailbreak_classification": {"cases": 400, "usable": 399}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 499}, "atbench500": {"cases": 500, "usable": 493}, "_segments": {"records": 5089, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "hi": {"jailbreak_classification": {"cases": 400, "usable": 397}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 499}, "atbench500": {"cases": 500, "usable": 487}, "_segments": {"records": 14759, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "vi": {"jailbreak_classification": {"cases": 400, "usable": 394}, "prompt_injections": {"cases": 116, "usable": 115}, "aegis2": {"cases": 500, "usable": 497}, "aegis2_response": {"cases": 500, "usable": 499}, "atbench500": {"cases": 500, "usable": 463}, "_segments": {"records": 5145, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "sw": {"jailbreak_classification": {"cases": 400, "usable": 398}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 499}, "atbench500": {"cases": 500, "usable": 484}, "_segments": {"records": 5333, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "es": {"jailbreak_classification": {"cases": 400, "usable": 396}, "prompt_injections": {"cases": 116, "usable": 113}, "aegis2": {"cases": 500, "usable": 499}, "aegis2_response": {"cases": 500, "usable": 500}, "atbench500": {"cases": 500, "usable": 451}, "_segments": {"records": 6637, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "fr": {"jailbreak_classification": {"cases": 400, "usable": 395}, "prompt_injections": {"cases": 116, "usable": 115}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 500}, "atbench500": {"cases": 500, "usable": 459}, "_segments": {"records": 5132, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "ru": {"jailbreak_classification": {"cases": 400, "usable": 399}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 500}, "atbench500": {"cases": 500, "usable": 490}, "_segments": {"records": 5092, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}} Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation. Checked / reported: 2026-10-20 · Samples: 499 Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
JevBench multilingual · Chinese · aegis2 · auroc	0.9205 coefficient	Setup & source JevBench multilingual · Chinese · aegis2 · auroc Mercury Decide, same named hosted route; Chinese; slice aegis2. {"answered": 1979, "cases": 1979, "complete": true} Full source observations: {"n": 499, "acc": 0.8437, "en_acc": 0.8397, "auroc": 0.9205, "en_auroc": 0.9276, "yes": 0.4729, "en_yes": 0.4649, "gold_yes": 0.501}. Translation provenance: {"zh": {"jailbreak_classification": {"cases": 400, "usable": 397}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 499}, "aegis2_response": {"cases": 500, "usable": 500}, "atbench500": {"cases": 500, "usable": 467}, "_segments": {"records": 5191, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "ar": {"jailbreak_classification": {"cases": 400, "usable": 399}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 499}, "atbench500": {"cases": 500, "usable": 493}, "_segments": {"records": 5089, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "hi": {"jailbreak_classification": {"cases": 400, "usable": 397}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 499}, "atbench500": {"cases": 500, "usable": 487}, "_segments": {"records": 14759, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "vi": {"jailbreak_classification": {"cases": 400, "usable": 394}, "prompt_injections": {"cases": 116, "usable": 115}, "aegis2": {"cases": 500, "usable": 497}, "aegis2_response": {"cases": 500, "usable": 499}, "atbench500": {"cases": 500, "usable": 463}, "_segments": {"records": 5145, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "sw": {"jailbreak_classification": {"cases": 400, "usable": 398}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 499}, "atbench500": {"cases": 500, "usable": 484}, "_segments": {"records": 5333, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "es": {"jailbreak_classification": {"cases": 400, "usable": 396}, "prompt_injections": {"cases": 116, "usable": 113}, "aegis2": {"cases": 500, "usable": 499}, "aegis2_response": {"cases": 500, "usable": 500}, "atbench500": {"cases": 500, "usable": 451}, "_segments": {"records": 6637, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "fr": {"jailbreak_classification": {"cases": 400, "usable": 395}, "prompt_injections": {"cases": 116, "usable": 115}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 500}, "atbench500": {"cases": 500, "usable": 459}, "_segments": {"records": 5132, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "ru": {"jailbreak_classification": {"cases": 400, "usable": 399}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 500}, "atbench500": {"cases": 500, "usable": 490}, "_segments": {"records": 5092, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}} Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation. Checked / reported: 2026-10-20 · Samples: 499 Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
JevBench multilingual · Chinese · aegis2 · en_acc	83.97%	Setup & source JevBench multilingual · Chinese · aegis2 · en_acc Mercury Decide, same named hosted route; Chinese; slice aegis2. {"answered": 1979, "cases": 1979, "complete": true} Full source observations: {"n": 499, "acc": 0.8437, "en_acc": 0.8397, "auroc": 0.9205, "en_auroc": 0.9276, "yes": 0.4729, "en_yes": 0.4649, "gold_yes": 0.501}. Translation provenance: {"zh": {"jailbreak_classification": {"cases": 400, "usable": 397}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 499}, "aegis2_response": {"cases": 500, "usable": 500}, "atbench500": {"cases": 500, "usable": 467}, "_segments": {"records": 5191, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "ar": {"jailbreak_classification": {"cases": 400, "usable": 399}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 499}, "atbench500": {"cases": 500, "usable": 493}, "_segments": {"records": 5089, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "hi": {"jailbreak_classification": {"cases": 400, "usable": 397}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 499}, "atbench500": {"cases": 500, "usable": 487}, "_segments": {"records": 14759, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "vi": {"jailbreak_classification": {"cases": 400, "usable": 394}, "prompt_injections": {"cases": 116, "usable": 115}, "aegis2": {"cases": 500, "usable": 497}, "aegis2_response": {"cases": 500, "usable": 499}, "atbench500": {"cases": 500, "usable": 463}, "_segments": {"records": 5145, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "sw": {"jailbreak_classification": {"cases": 400, "usable": 398}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 499}, "atbench500": {"cases": 500, "usable": 484}, "_segments": {"records": 5333, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "es": {"jailbreak_classification": {"cases": 400, "usable": 396}, "prompt_injections": {"cases": 116, "usable": 113}, "aegis2": {"cases": 500, "usable": 499}, "aegis2_response": {"cases": 500, "usable": 500}, "atbench500": {"cases": 500, "usable": 451}, "_segments": {"records": 6637, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "fr": {"jailbreak_classification": {"cases": 400, "usable": 395}, "prompt_injections": {"cases": 116, "usable": 115}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 500}, "atbench500": {"cases": 500, "usable": 459}, "_segments": {"records": 5132, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "ru": {"jailbreak_classification": {"cases": 400, "usable": 399}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 500}, "atbench500": {"cases": 500, "usable": 490}, "_segments": {"records": 5092, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}} Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation. Checked / reported: 2026-10-20 · Samples: 499 Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
JevBench multilingual · Chinese · aegis2 · en_auroc	0.9276 coefficient	Setup & source JevBench multilingual · Chinese · aegis2 · en_auroc Mercury Decide, same named hosted route; Chinese; slice aegis2. {"answered": 1979, "cases": 1979, "complete": true} Full source observations: {"n": 499, "acc": 0.8437, "en_acc": 0.8397, "auroc": 0.9205, "en_auroc": 0.9276, "yes": 0.4729, "en_yes": 0.4649, "gold_yes": 0.501}. Translation provenance: {"zh": {"jailbreak_classification": {"cases": 400, "usable": 397}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 499}, "aegis2_response": {"cases": 500, "usable": 500}, "atbench500": {"cases": 500, "usable": 467}, "_segments": {"records": 5191, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "ar": {"jailbreak_classification": {"cases": 400, "usable": 399}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 499}, "atbench500": {"cases": 500, "usable": 493}, "_segments": {"records": 5089, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "hi": {"jailbreak_classification": {"cases": 400, "usable": 397}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 499}, "atbench500": {"cases": 500, "usable": 487}, "_segments": {"records": 14759, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "vi": {"jailbreak_classification": {"cases": 400, "usable": 394}, "prompt_injections": {"cases": 116, "usable": 115}, "aegis2": {"cases": 500, "usable": 497}, "aegis2_response": {"cases": 500, "usable": 499}, "atbench500": {"cases": 500, "usable": 463}, "_segments": {"records": 5145, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "sw": {"jailbreak_classification": {"cases": 400, "usable": 398}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 499}, "atbench500": {"cases": 500, "usable": 484}, "_segments": {"records": 5333, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "es": {"jailbreak_classification": {"cases": 400, "usable": 396}, "prompt_injections": {"cases": 116, "usable": 113}, "aegis2": {"cases": 500, "usable": 499}, "aegis2_response": {"cases": 500, "usable": 500}, "atbench500": {"cases": 500, "usable": 451}, "_segments": {"records": 6637, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "fr": {"jailbreak_classification": {"cases": 400, "usable": 395}, "prompt_injections": {"cases": 116, "usable": 115}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 500}, "atbench500": {"cases": 500, "usable": 459}, "_segments": {"records": 5132, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "ru": {"jailbreak_classification": {"cases": 400, "usable": 399}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 500}, "atbench500": {"cases": 500, "usable": 490}, "_segments": {"records": 5092, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}} Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation. Checked / reported: 2026-10-20 · Samples: 499 Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
JevBench multilingual · Chinese · aegis2 · yes	47.29%	Setup & source JevBench multilingual · Chinese · aegis2 · yes Mercury Decide, same named hosted route; Chinese; slice aegis2. {"answered": 1979, "cases": 1979, "complete": true} Full source observations: {"n": 499, "acc": 0.8437, "en_acc": 0.8397, "auroc": 0.9205, "en_auroc": 0.9276, "yes": 0.4729, "en_yes": 0.4649, "gold_yes": 0.501}. Translation provenance: {"zh": {"jailbreak_classification": {"cases": 400, "usable": 397}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 499}, "aegis2_response": {"cases": 500, "usable": 500}, "atbench500": {"cases": 500, "usable": 467}, "_segments": {"records": 5191, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "ar": {"jailbreak_classification": {"cases": 400, "usable": 399}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 499}, "atbench500": {"cases": 500, "usable": 493}, "_segments": {"records": 5089, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "hi": {"jailbreak_classification": {"cases": 400, "usable": 397}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 499}, "atbench500": {"cases": 500, "usable": 487}, "_segments": {"records": 14759, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "vi": {"jailbreak_classification": {"cases": 400, "usable": 394}, "prompt_injections": {"cases": 116, "usable": 115}, "aegis2": {"cases": 500, "usable": 497}, "aegis2_response": {"cases": 500, "usable": 499}, "atbench500": {"cases": 500, "usable": 463}, "_segments": {"records": 5145, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "sw": {"jailbreak_classification": {"cases": 400, "usable": 398}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 499}, "atbench500": {"cases": 500, "usable": 484}, "_segments": {"records": 5333, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "es": {"jailbreak_classification": {"cases": 400, "usable": 396}, "prompt_injections": {"cases": 116, "usable": 113}, "aegis2": {"cases": 500, "usable": 499}, "aegis2_response": {"cases": 500, "usable": 500}, "atbench500": {"cases": 500, "usable": 451}, "_segments": {"records": 6637, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "fr": {"jailbreak_classification": {"cases": 400, "usable": 395}, "prompt_injections": {"cases": 116, "usable": 115}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 500}, "atbench500": {"cases": 500, "usable": 459}, "_segments": {"records": 5132, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "ru": {"jailbreak_classification": {"cases": 400, "usable": 399}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 500}, "atbench500": {"cases": 500, "usable": 490}, "_segments": {"records": 5092, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}} Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation. Checked / reported: 2026-10-20 · Samples: 499 Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
JevBench multilingual · Chinese · aegis2 · en_yes	46.49%	Setup & source JevBench multilingual · Chinese · aegis2 · en_yes Mercury Decide, same named hosted route; Chinese; slice aegis2. {"answered": 1979, "cases": 1979, "complete": true} Full source observations: {"n": 499, "acc": 0.8437, "en_acc": 0.8397, "auroc": 0.9205, "en_auroc": 0.9276, "yes": 0.4729, "en_yes": 0.4649, "gold_yes": 0.501}. Translation provenance: {"zh": {"jailbreak_classification": {"cases": 400, "usable": 397}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 499}, "aegis2_response": {"cases": 500, "usable": 500}, "atbench500": {"cases": 500, "usable": 467}, "_segments": {"records": 5191, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "ar": {"jailbreak_classification": {"cases": 400, "usable": 399}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 499}, "atbench500": {"cases": 500, "usable": 493}, "_segments": {"records": 5089, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "hi": {"jailbreak_classification": {"cases": 400, "usable": 397}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 499}, "atbench500": {"cases": 500, "usable": 487}, "_segments": {"records": 14759, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "vi": {"jailbreak_classification": {"cases": 400, "usable": 394}, "prompt_injections": {"cases": 116, "usable": 115}, "aegis2": {"cases": 500, "usable": 497}, "aegis2_response": {"cases": 500, "usable": 499}, "atbench500": {"cases": 500, "usable": 463}, "_segments": {"records": 5145, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "sw": {"jailbreak_classification": {"cases": 400, "usable": 398}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 499}, "atbench500": {"cases": 500, "usable": 484}, "_segments": {"records": 5333, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "es": {"jailbreak_classification": {"cases": 400, "usable": 396}, "prompt_injections": {"cases": 116, "usable": 113}, "aegis2": {"cases": 500, "usable": 499}, "aegis2_response": {"cases": 500, "usable": 500}, "atbench500": {"cases": 500, "usable": 451}, "_segments": {"records": 6637, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "fr": {"jailbreak_classification": {"cases": 400, "usable": 395}, "prompt_injections": {"cases": 116, "usable": 115}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 500}, "atbench500": {"cases": 500, "usable": 459}, "_segments": {"records": 5132, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "ru": {"jailbreak_classification": {"cases": 400, "usable": 399}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 500}, "atbench500": {"cases": 500, "usable": 490}, "_segments": {"records": 5092, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}} Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation. Checked / reported: 2026-10-20 · Samples: 499 Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
JevBench multilingual · Chinese · aegis2_response · acc	75.80%	Setup & source JevBench multilingual · Chinese · aegis2_response · acc Mercury Decide, same named hosted route; Chinese; slice aegis2_response. {"answered": 1979, "cases": 1979, "complete": true} Full source observations: {"n": 500, "acc": 0.758, "en_acc": 0.772, "auroc": 0.8726, "en_auroc": 0.8876, "yes": 0.37, "en_yes": 0.352, "gold_yes": 0.5}. Translation provenance: {"zh": {"jailbreak_classification": {"cases": 400, "usable": 397}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 499}, "aegis2_response": {"cases": 500, "usable": 500}, "atbench500": {"cases": 500, "usable": 467}, "_segments": {"records": 5191, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "ar": {"jailbreak_classification": {"cases": 400, "usable": 399}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 499}, "atbench500": {"cases": 500, "usable": 493}, "_segments": {"records": 5089, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "hi": {"jailbreak_classification": {"cases": 400, "usable": 397}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 499}, "atbench500": {"cases": 500, "usable": 487}, "_segments": {"records": 14759, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "vi": {"jailbreak_classification": {"cases": 400, "usable": 394}, "prompt_injections": {"cases": 116, "usable": 115}, "aegis2": {"cases": 500, "usable": 497}, "aegis2_response": {"cases": 500, "usable": 499}, "atbench500": {"cases": 500, "usable": 463}, "_segments": {"records": 5145, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "sw": {"jailbreak_classification": {"cases": 400, "usable": 398}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 499}, "atbench500": {"cases": 500, "usable": 484}, "_segments": {"records": 5333, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "es": {"jailbreak_classification": {"cases": 400, "usable": 396}, "prompt_injections": {"cases": 116, "usable": 113}, "aegis2": {"cases": 500, "usable": 499}, "aegis2_response": {"cases": 500, "usable": 500}, "atbench500": {"cases": 500, "usable": 451}, "_segments": {"records": 6637, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "fr": {"jailbreak_classification": {"cases": 400, "usable": 395}, "prompt_injections": {"cases": 116, "usable": 115}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 500}, "atbench500": {"cases": 500, "usable": 459}, "_segments": {"records": 5132, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "ru": {"jailbreak_classification": {"cases": 400, "usable": 399}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 500}, "atbench500": {"cases": 500, "usable": 490}, "_segments": {"records": 5092, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}} Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation. Checked / reported: 2026-10-20 · Samples: 500 Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
JevBench multilingual · Chinese · aegis2_response · auroc	0.8726 coefficient	Setup & source JevBench multilingual · Chinese · aegis2_response · auroc Mercury Decide, same named hosted route; Chinese; slice aegis2_response. {"answered": 1979, "cases": 1979, "complete": true} Full source observations: {"n": 500, "acc": 0.758, "en_acc": 0.772, "auroc": 0.8726, "en_auroc": 0.8876, "yes": 0.37, "en_yes": 0.352, "gold_yes": 0.5}. Translation provenance: {"zh": {"jailbreak_classification": {"cases": 400, "usable": 397}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 499}, "aegis2_response": {"cases": 500, "usable": 500}, "atbench500": {"cases": 500, "usable": 467}, "_segments": {"records": 5191, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "ar": {"jailbreak_classification": {"cases": 400, "usable": 399}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 499}, "atbench500": {"cases": 500, "usable": 493}, "_segments": {"records": 5089, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "hi": {"jailbreak_classification": {"cases": 400, "usable": 397}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 499}, "atbench500": {"cases": 500, "usable": 487}, "_segments": {"records": 14759, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "vi": {"jailbreak_classification": {"cases": 400, "usable": 394}, "prompt_injections": {"cases": 116, "usable": 115}, "aegis2": {"cases": 500, "usable": 497}, "aegis2_response": {"cases": 500, "usable": 499}, "atbench500": {"cases": 500, "usable": 463}, "_segments": {"records": 5145, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "sw": {"jailbreak_classification": {"cases": 400, "usable": 398}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 499}, "atbench500": {"cases": 500, "usable": 484}, "_segments": {"records": 5333, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "es": {"jailbreak_classification": {"cases": 400, "usable": 396}, "prompt_injections": {"cases": 116, "usable": 113}, "aegis2": {"cases": 500, "usable": 499}, "aegis2_response": {"cases": 500, "usable": 500}, "atbench500": {"cases": 500, "usable": 451}, "_segments": {"records": 6637, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "fr": {"jailbreak_classification": {"cases": 400, "usable": 395}, "prompt_injections": {"cases": 116, "usable": 115}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 500}, "atbench500": {"cases": 500, "usable": 459}, "_segments": {"records": 5132, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "ru": {"jailbreak_classification": {"cases": 400, "usable": 399}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 500}, "atbench500": {"cases": 500, "usable": 490}, "_segments": {"records": 5092, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}} Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation. Checked / reported: 2026-10-20 · Samples: 500 Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
JevBench multilingual · Chinese · aegis2_response · en_acc	77.20%	Setup & source JevBench multilingual · Chinese · aegis2_response · en_acc Mercury Decide, same named hosted route; Chinese; slice aegis2_response. {"answered": 1979, "cases": 1979, "complete": true} Full source observations: {"n": 500, "acc": 0.758, "en_acc": 0.772, "auroc": 0.8726, "en_auroc": 0.8876, "yes": 0.37, "en_yes": 0.352, "gold_yes": 0.5}. Translation provenance: {"zh": {"jailbreak_classification": {"cases": 400, "usable": 397}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 499}, "aegis2_response": {"cases": 500, "usable": 500}, "atbench500": {"cases": 500, "usable": 467}, "_segments": {"records": 5191, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "ar": {"jailbreak_classification": {"cases": 400, "usable": 399}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 499}, "atbench500": {"cases": 500, "usable": 493}, "_segments": {"records": 5089, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "hi": {"jailbreak_classification": {"cases": 400, "usable": 397}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 499}, "atbench500": {"cases": 500, "usable": 487}, "_segments": {"records": 14759, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "vi": {"jailbreak_classification": {"cases": 400, "usable": 394}, "prompt_injections": {"cases": 116, "usable": 115}, "aegis2": {"cases": 500, "usable": 497}, "aegis2_response": {"cases": 500, "usable": 499}, "atbench500": {"cases": 500, "usable": 463}, "_segments": {"records": 5145, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "sw": {"jailbreak_classification": {"cases": 400, "usable": 398}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 499}, "atbench500": {"cases": 500, "usable": 484}, "_segments": {"records": 5333, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "es": {"jailbreak_classification": {"cases": 400, "usable": 396}, "prompt_injections": {"cases": 116, "usable": 113}, "aegis2": {"cases": 500, "usable": 499}, "aegis2_response": {"cases": 500, "usable": 500}, "atbench500": {"cases": 500, "usable": 451}, "_segments": {"records": 6637, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "fr": {"jailbreak_classification": {"cases": 400, "usable": 395}, "prompt_injections": {"cases": 116, "usable": 115}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 500}, "atbench500": {"cases": 500, "usable": 459}, "_segments": {"records": 5132, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "ru": {"jailbreak_classification": {"cases": 400, "usable": 399}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 500}, "atbench500": {"cases": 500, "usable": 490}, "_segments": {"records": 5092, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}} Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation. Checked / reported: 2026-10-20 · Samples: 500 Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
JevBench multilingual · Chinese · aegis2_response · en_auroc	0.8876 coefficient	Setup & source JevBench multilingual · Chinese · aegis2_response · en_auroc Mercury Decide, same named hosted route; Chinese; slice aegis2_response. {"answered": 1979, "cases": 1979, "complete": true} Full source observations: {"n": 500, "acc": 0.758, "en_acc": 0.772, "auroc": 0.8726, "en_auroc": 0.8876, "yes": 0.37, "en_yes": 0.352, "gold_yes": 0.5}. Translation provenance: {"zh": {"jailbreak_classification": {"cases": 400, "usable": 397}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 499}, "aegis2_response": {"cases": 500, "usable": 500}, "atbench500": {"cases": 500, "usable": 467}, "_segments": {"records": 5191, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "ar": {"jailbreak_classification": {"cases": 400, "usable": 399}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 499}, "atbench500": {"cases": 500, "usable": 493}, "_segments": {"records": 5089, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "hi": {"jailbreak_classification": {"cases": 400, "usable": 397}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 499}, "atbench500": {"cases": 500, "usable": 487}, "_segments": {"records": 14759, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "vi": {"jailbreak_classification": {"cases": 400, "usable": 394}, "prompt_injections": {"cases": 116, "usable": 115}, "aegis2": {"cases": 500, "usable": 497}, "aegis2_response": {"cases": 500, "usable": 499}, "atbench500": {"cases": 500, "usable": 463}, "_segments": {"records": 5145, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "sw": {"jailbreak_classification": {"cases": 400, "usable": 398}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 499}, "atbench500": {"cases": 500, "usable": 484}, "_segments": {"records": 5333, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "es": {"jailbreak_classification": {"cases": 400, "usable": 396}, "prompt_injections": {"cases": 116, "usable": 113}, "aegis2": {"cases": 500, "usable": 499}, "aegis2_response": {"cases": 500, "usable": 500}, "atbench500": {"cases": 500, "usable": 451}, "_segments": {"records": 6637, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "fr": {"jailbreak_classification": {"cases": 400, "usable": 395}, "prompt_injections": {"cases": 116, "usable": 115}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 500}, "atbench500": {"cases": 500, "usable": 459}, "_segments": {"records": 5132, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "ru": {"jailbreak_classification": {"cases": 400, "usable": 399}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 500}, "atbench500": {"cases": 500, "usable": 490}, "_segments": {"records": 5092, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}} Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation. Checked / reported: 2026-10-20 · Samples: 500 Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
JevBench multilingual · Chinese · aegis2_response · yes	37.00%	Setup & source JevBench multilingual · Chinese · aegis2_response · yes Mercury Decide, same named hosted route; Chinese; slice aegis2_response. {"answered": 1979, "cases": 1979, "complete": true} Full source observations: {"n": 500, "acc": 0.758, "en_acc": 0.772, "auroc": 0.8726, "en_auroc": 0.8876, "yes": 0.37, "en_yes": 0.352, "gold_yes": 0.5}. Translation provenance: {"zh": {"jailbreak_classification": {"cases": 400, "usable": 397}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 499}, "aegis2_response": {"cases": 500, "usable": 500}, "atbench500": {"cases": 500, "usable": 467}, "_segments": {"records": 5191, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "ar": {"jailbreak_classification": {"cases": 400, "usable": 399}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 499}, "atbench500": {"cases": 500, "usable": 493}, "_segments": {"records": 5089, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "hi": {"jailbreak_classification": {"cases": 400, "usable": 397}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 499}, "atbench500": {"cases": 500, "usable": 487}, "_segments": {"records": 14759, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "vi": {"jailbreak_classification": {"cases": 400, "usable": 394}, "prompt_injections": {"cases": 116, "usable": 115}, "aegis2": {"cases": 500, "usable": 497}, "aegis2_response": {"cases": 500, "usable": 499}, "atbench500": {"cases": 500, "usable": 463}, "_segments": {"records": 5145, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "sw": {"jailbreak_classification": {"cases": 400, "usable": 398}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 499}, "atbench500": {"cases": 500, "usable": 484}, "_segments": {"records": 5333, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "es": {"jailbreak_classification": {"cases": 400, "usable": 396}, "prompt_injections": {"cases": 116, "usable": 113}, "aegis2": {"cases": 500, "usable": 499}, "aegis2_response": {"cases": 500, "usable": 500}, "atbench500": {"cases": 500, "usable": 451}, "_segments": {"records": 6637, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "fr": {"jailbreak_classification": {"cases": 400, "usable": 395}, "prompt_injections": {"cases": 116, "usable": 115}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 500}, "atbench500": {"cases": 500, "usable": 459}, "_segments": {"records": 5132, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "ru": {"jailbreak_classification": {"cases": 400, "usable": 399}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 500}, "atbench500": {"cases": 500, "usable": 490}, "_segments": {"records": 5092, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}} Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation. Checked / reported: 2026-10-20 · Samples: 500 Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
JevBench multilingual · Chinese · aegis2_response · en_yes	35.20%	Setup & source JevBench multilingual · Chinese · aegis2_response · en_yes Mercury Decide, same named hosted route; Chinese; slice aegis2_response. {"answered": 1979, "cases": 1979, "complete": true} Full source observations: {"n": 500, "acc": 0.758, "en_acc": 0.772, "auroc": 0.8726, "en_auroc": 0.8876, "yes": 0.37, "en_yes": 0.352, "gold_yes": 0.5}. Translation provenance: {"zh": {"jailbreak_classification": {"cases": 400, "usable": 397}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 499}, "aegis2_response": {"cases": 500, "usable": 500}, "atbench500": {"cases": 500, "usable": 467}, "_segments": {"records": 5191, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "ar": {"jailbreak_classification": {"cases": 400, "usable": 399}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 499}, "atbench500": {"cases": 500, "usable": 493}, "_segments": {"records": 5089, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "hi": {"jailbreak_classification": {"cases": 400, "usable": 397}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 499}, "atbench500": {"cases": 500, "usable": 487}, "_segments": {"records": 14759, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "vi": {"jailbreak_classification": {"cases": 400, "usable": 394}, "prompt_injections": {"cases": 116, "usable": 115}, "aegis2": {"cases": 500, "usable": 497}, "aegis2_response": {"cases": 500, "usable": 499}, "atbench500": {"cases": 500, "usable": 463}, "_segments": {"records": 5145, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "sw": {"jailbreak_classification": {"cases": 400, "usable": 398}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 499}, "atbench500": {"cases": 500, "usable": 484}, "_segments": {"records": 5333, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "es": {"jailbreak_classification": {"cases": 400, "usable": 396}, "prompt_injections": {"cases": 116, "usable": 113}, "aegis2": {"cases": 500, "usable": 499}, "aegis2_response": {"cases": 500, "usable": 500}, "atbench500": {"cases": 500, "usable": 451}, "_segments": {"records": 6637, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "fr": {"jailbreak_classification": {"cases": 400, "usable": 395}, "prompt_injections": {"cases": 116, "usable": 115}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 500}, "atbench500": {"cases": 500, "usable": 459}, "_segments": {"records": 5132, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "ru": {"jailbreak_classification": {"cases": 400, "usable": 399}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 500}, "atbench500": {"cases": 500, "usable": 490}, "_segments": {"records": 5092, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}} Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation. Checked / reported: 2026-10-20 · Samples: 500 Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
JevBench multilingual · Chinese · atbench500 · acc	67.02%	Setup & source JevBench multilingual · Chinese · atbench500 · acc Mercury Decide, same named hosted route; Chinese; slice atbench500. {"answered": 1979, "cases": 1979, "complete": true} Full source observations: {"n": 467, "acc": 0.6702, "en_acc": 0.6874, "auroc": 0.9865, "en_auroc": 0.9924, "yes": 0.1627, "en_yes": 0.1799, "gold_yes": 0.4925}. Translation provenance: {"zh": {"jailbreak_classification": {"cases": 400, "usable": 397}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 499}, "aegis2_response": {"cases": 500, "usable": 500}, "atbench500": {"cases": 500, "usable": 467}, "_segments": {"records": 5191, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "ar": {"jailbreak_classification": {"cases": 400, "usable": 399}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 499}, "atbench500": {"cases": 500, "usable": 493}, "_segments": {"records": 5089, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "hi": {"jailbreak_classification": {"cases": 400, "usable": 397}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 499}, "atbench500": {"cases": 500, "usable": 487}, "_segments": {"records": 14759, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "vi": {"jailbreak_classification": {"cases": 400, "usable": 394}, "prompt_injections": {"cases": 116, "usable": 115}, "aegis2": {"cases": 500, "usable": 497}, "aegis2_response": {"cases": 500, "usable": 499}, "atbench500": {"cases": 500, "usable": 463}, "_segments": {"records": 5145, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "sw": {"jailbreak_classification": {"cases": 400, "usable": 398}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 499}, "atbench500": {"cases": 500, "usable": 484}, "_segments": {"records": 5333, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "es": {"jailbreak_classification": {"cases": 400, "usable": 396}, "prompt_injections": {"cases": 116, "usable": 113}, "aegis2": {"cases": 500, "usable": 499}, "aegis2_response": {"cases": 500, "usable": 500}, "atbench500": {"cases": 500, "usable": 451}, "_segments": {"records": 6637, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "fr": {"jailbreak_classification": {"cases": 400, "usable": 395}, "prompt_injections": {"cases": 116, "usable": 115}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 500}, "atbench500": {"cases": 500, "usable": 459}, "_segments": {"records": 5132, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "ru": {"jailbreak_classification": {"cases": 400, "usable": 399}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 500}, "atbench500": {"cases": 500, "usable": 490}, "_segments": {"records": 5092, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}} Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation. Checked / reported: 2026-10-20 · Samples: 467 Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
JevBench multilingual · Chinese · atbench500 · auroc	0.9865 coefficient	Setup & source JevBench multilingual · Chinese · atbench500 · auroc Mercury Decide, same named hosted route; Chinese; slice atbench500. {"answered": 1979, "cases": 1979, "complete": true} Full source observations: {"n": 467, "acc": 0.6702, "en_acc": 0.6874, "auroc": 0.9865, "en_auroc": 0.9924, "yes": 0.1627, "en_yes": 0.1799, "gold_yes": 0.4925}. Translation provenance: {"zh": {"jailbreak_classification": {"cases": 400, "usable": 397}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 499}, "aegis2_response": {"cases": 500, "usable": 500}, "atbench500": {"cases": 500, "usable": 467}, "_segments": {"records": 5191, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "ar": {"jailbreak_classification": {"cases": 400, "usable": 399}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 499}, "atbench500": {"cases": 500, "usable": 493}, "_segments": {"records": 5089, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "hi": {"jailbreak_classification": {"cases": 400, "usable": 397}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 499}, "atbench500": {"cases": 500, "usable": 487}, "_segments": {"records": 14759, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "vi": {"jailbreak_classification": {"cases": 400, "usable": 394}, "prompt_injections": {"cases": 116, "usable": 115}, "aegis2": {"cases": 500, "usable": 497}, "aegis2_response": {"cases": 500, "usable": 499}, "atbench500": {"cases": 500, "usable": 463}, "_segments": {"records": 5145, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "sw": {"jailbreak_classification": {"cases": 400, "usable": 398}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 499}, "atbench500": {"cases": 500, "usable": 484}, "_segments": {"records": 5333, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "es": {"jailbreak_classification": {"cases": 400, "usable": 396}, "prompt_injections": {"cases": 116, "usable": 113}, "aegis2": {"cases": 500, "usable": 499}, "aegis2_response": {"cases": 500, "usable": 500}, "atbench500": {"cases": 500, "usable": 451}, "_segments": {"records": 6637, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "fr": {"jailbreak_classification": {"cases": 400, "usable": 395}, "prompt_injections": {"cases": 116, "usable": 115}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 500}, "atbench500": {"cases": 500, "usable": 459}, "_segments": {"records": 5132, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "ru": {"jailbreak_classification": {"cases": 400, "usable": 399}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 500}, "atbench500": {"cases": 500, "usable": 490}, "_segments": {"records": 5092, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}} Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation. Checked / reported: 2026-10-20 · Samples: 467 Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
JevBench multilingual · Chinese · atbench500 · en_acc	68.74%	Setup & source JevBench multilingual · Chinese · atbench500 · en_acc Mercury Decide, same named hosted route; Chinese; slice atbench500. {"answered": 1979, "cases": 1979, "complete": true} Full source observations: {"n": 467, "acc": 0.6702, "en_acc": 0.6874, "auroc": 0.9865, "en_auroc": 0.9924, "yes": 0.1627, "en_yes": 0.1799, "gold_yes": 0.4925}. Translation provenance: {"zh": {"jailbreak_classification": {"cases": 400, "usable": 397}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 499}, "aegis2_response": {"cases": 500, "usable": 500}, "atbench500": {"cases": 500, "usable": 467}, "_segments": {"records": 5191, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "ar": {"jailbreak_classification": {"cases": 400, "usable": 399}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 499}, "atbench500": {"cases": 500, "usable": 493}, "_segments": {"records": 5089, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "hi": {"jailbreak_classification": {"cases": 400, "usable": 397}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 499}, "atbench500": {"cases": 500, "usable": 487}, "_segments": {"records": 14759, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "vi": {"jailbreak_classification": {"cases": 400, "usable": 394}, "prompt_injections": {"cases": 116, "usable": 115}, "aegis2": {"cases": 500, "usable": 497}, "aegis2_response": {"cases": 500, "usable": 499}, "atbench500": {"cases": 500, "usable": 463}, "_segments": {"records": 5145, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "sw": {"jailbreak_classification": {"cases": 400, "usable": 398}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 499}, "atbench500": {"cases": 500, "usable": 484}, "_segments": {"records": 5333, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "es": {"jailbreak_classification": {"cases": 400, "usable": 396}, "prompt_injections": {"cases": 116, "usable": 113}, "aegis2": {"cases": 500, "usable": 499}, "aegis2_response": {"cases": 500, "usable": 500}, "atbench500": {"cases": 500, "usable": 451}, "_segments": {"records": 6637, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "fr": {"jailbreak_classification": {"cases": 400, "usable": 395}, "prompt_injections": {"cases": 116, "usable": 115}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 500}, "atbench500": {"cases": 500, "usable": 459}, "_segments": {"records": 5132, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "ru": {"jailbreak_classification": {"cases": 400, "usable": 399}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 500}, "atbench500": {"cases": 500, "usable": 490}, "_segments": {"records": 5092, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}} Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation. Checked / reported: 2026-10-20 · Samples: 467 Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
JevBench multilingual · Chinese · atbench500 · en_auroc	0.9924 coefficient	Setup & source JevBench multilingual · Chinese · atbench500 · en_auroc Mercury Decide, same named hosted route; Chinese; slice atbench500. {"answered": 1979, "cases": 1979, "complete": true} Full source observations: {"n": 467, "acc": 0.6702, "en_acc": 0.6874, "auroc": 0.9865, "en_auroc": 0.9924, "yes": 0.1627, "en_yes": 0.1799, "gold_yes": 0.4925}. Translation provenance: {"zh": {"jailbreak_classification": {"cases": 400, "usable": 397}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 499}, "aegis2_response": {"cases": 500, "usable": 500}, "atbench500": {"cases": 500, "usable": 467}, "_segments": {"records": 5191, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "ar": {"jailbreak_classification": {"cases": 400, "usable": 399}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 499}, "atbench500": {"cases": 500, "usable": 493}, "_segments": {"records": 5089, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "hi": {"jailbreak_classification": {"cases": 400, "usable": 397}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 499}, "atbench500": {"cases": 500, "usable": 487}, "_segments": {"records": 14759, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "vi": {"jailbreak_classification": {"cases": 400, "usable": 394}, "prompt_injections": {"cases": 116, "usable": 115}, "aegis2": {"cases": 500, "usable": 497}, "aegis2_response": {"cases": 500, "usable": 499}, "atbench500": {"cases": 500, "usable": 463}, "_segments": {"records": 5145, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "sw": {"jailbreak_classification": {"cases": 400, "usable": 398}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 499}, "atbench500": {"cases": 500, "usable": 484}, "_segments": {"records": 5333, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "es": {"jailbreak_classification": {"cases": 400, "usable": 396}, "prompt_injections": {"cases": 116, "usable": 113}, "aegis2": {"cases": 500, "usable": 499}, "aegis2_response": {"cases": 500, "usable": 500}, "atbench500": {"cases": 500, "usable": 451}, "_segments": {"records": 6637, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "fr": {"jailbreak_classification": {"cases": 400, "usable": 395}, "prompt_injections": {"cases": 116, "usable": 115}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 500}, "atbench500": {"cases": 500, "usable": 459}, "_segments": {"records": 5132, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "ru": {"jailbreak_classification": {"cases": 400, "usable": 399}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 500}, "atbench500": {"cases": 500, "usable": 490}, "_segments": {"records": 5092, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}} Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation. Checked / reported: 2026-10-20 · Samples: 467 Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
JevBench multilingual · Chinese · atbench500 · yes	16.27%	Setup & source JevBench multilingual · Chinese · atbench500 · yes Mercury Decide, same named hosted route; Chinese; slice atbench500. {"answered": 1979, "cases": 1979, "complete": true} Full source observations: {"n": 467, "acc": 0.6702, "en_acc": 0.6874, "auroc": 0.9865, "en_auroc": 0.9924, "yes": 0.1627, "en_yes": 0.1799, "gold_yes": 0.4925}. Translation provenance: {"zh": {"jailbreak_classification": {"cases": 400, "usable": 397}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 499}, "aegis2_response": {"cases": 500, "usable": 500}, "atbench500": {"cases": 500, "usable": 467}, "_segments": {"records": 5191, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "ar": {"jailbreak_classification": {"cases": 400, "usable": 399}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 499}, "atbench500": {"cases": 500, "usable": 493}, "_segments": {"records": 5089, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "hi": {"jailbreak_classification": {"cases": 400, "usable": 397}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 499}, "atbench500": {"cases": 500, "usable": 487}, "_segments": {"records": 14759, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "vi": {"jailbreak_classification": {"cases": 400, "usable": 394}, "prompt_injections": {"cases": 116, "usable": 115}, "aegis2": {"cases": 500, "usable": 497}, "aegis2_response": {"cases": 500, "usable": 499}, "atbench500": {"cases": 500, "usable": 463}, "_segments": {"records": 5145, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "sw": {"jailbreak_classification": {"cases": 400, "usable": 398}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 499}, "atbench500": {"cases": 500, "usable": 484}, "_segments": {"records": 5333, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "es": {"jailbreak_classification": {"cases": 400, "usable": 396}, "prompt_injections": {"cases": 116, "usable": 113}, "aegis2": {"cases": 500, "usable": 499}, "aegis2_response": {"cases": 500, "usable": 500}, "atbench500": {"cases": 500, "usable": 451}, "_segments": {"records": 6637, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "fr": {"jailbreak_classification": {"cases": 400, "usable": 395}, "prompt_injections": {"cases": 116, "usable": 115}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 500}, "atbench500": {"cases": 500, "usable": 459}, "_segments": {"records": 5132, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "ru": {"jailbreak_classification": {"cases": 400, "usable": 399}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 500}, "atbench500": {"cases": 500, "usable": 490}, "_segments": {"records": 5092, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}} Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation. Checked / reported: 2026-10-20 · Samples: 467 Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
JevBench multilingual · Chinese · atbench500 · en_yes	17.99%	Setup & source JevBench multilingual · Chinese · atbench500 · en_yes Mercury Decide, same named hosted route; Chinese; slice atbench500. {"answered": 1979, "cases": 1979, "complete": true} Full source observations: {"n": 467, "acc": 0.6702, "en_acc": 0.6874, "auroc": 0.9865, "en_auroc": 0.9924, "yes": 0.1627, "en_yes": 0.1799, "gold_yes": 0.4925}. Translation provenance: {"zh": {"jailbreak_classification": {"cases": 400, "usable": 397}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 499}, "aegis2_response": {"cases": 500, "usable": 500}, "atbench500": {"cases": 500, "usable": 467}, "_segments": {"records": 5191, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "ar": {"jailbreak_classification": {"cases": 400, "usable": 399}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 499}, "atbench500": {"cases": 500, "usable": 493}, "_segments": {"records": 5089, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "hi": {"jailbreak_classification": {"cases": 400, "usable": 397}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 499}, "atbench500": {"cases": 500, "usable": 487}, "_segments": {"records": 14759, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "vi": {"jailbreak_classification": {"cases": 400, "usable": 394}, "prompt_injections": {"cases": 116, "usable": 115}, "aegis2": {"cases": 500, "usable": 497}, "aegis2_response": {"cases": 500, "usable": 499}, "atbench500": {"cases": 500, "usable": 463}, "_segments": {"records": 5145, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "sw": {"jailbreak_classification": {"cases": 400, "usable": 398}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 499}, "atbench500": {"cases": 500, "usable": 484}, "_segments": {"records": 5333, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "es": {"jailbreak_classification": {"cases": 400, "usable": 396}, "prompt_injections": {"cases": 116, "usable": 113}, "aegis2": {"cases": 500, "usable": 499}, "aegis2_response": {"cases": 500, "usable": 500}, "atbench500": {"cases": 500, "usable": 451}, "_segments": {"records": 6637, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "fr": {"jailbreak_classification": {"cases": 400, "usable": 395}, "prompt_injections": {"cases": 116, "usable": 115}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 500}, "atbench500": {"cases": 500, "usable": 459}, "_segments": {"records": 5132, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}, "ru": {"jailbreak_classification": {"cases": 400, "usable": 399}, "prompt_injections": {"cases": 116, "usable": 116}, "aegis2": {"cases": 500, "usable": 500}, "aegis2_response": {"cases": 500, "usable": 500}, "atbench500": {"cases": 500, "usable": 490}, "_segments": {"records": 5092, "translator": ["gpt-6-astra"], "reasoning_effort": ["low"]}}} Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation. Checked / reported: 2026-10-20 · Samples: 467 Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained Result source · Benchmark source

Mercury Decide · Independent · Vals AI
Evaluation	Result	Evidence
Vals Mercury Decide study · Claim verification · all · accuracy	97.750%	Setup & source Vals Mercury Decide study · Claim verification · all · accuracy Vals8October2026 Mercury Decide direct Inception API; no version pin; one call/item, no malformed-response retries; same fixed instructions/labels as Jev study. Claims400SEC10-K items from48companies; LegalBench12preregistered subtasks×33questions. Historical$0.021/M input tariff; costs are usage×listedprice, not verified invoices. Routing threshold fit on30%, frozen before held-out70% (265items). Primary chart data module decideStudy.CSyRc3i6.js, parsed as literals without executing source code. Source decimals retained. ECE is a native fraction converted to percentage points; no mixture with Brier or ModelTested confidence score. Served checkpoint SHA/run dates unpublished; study publication8October is not all call execution dates. Primary chart data module decideStudy.CSyRc3i6.js, parsed as literals without executing source code. Source decimals retained. ECE is a native fraction converted to percentage points; no mixture with Brier or ModelTested confidence score. Served checkpoint SHA/run dates unpublished; study publication8October is not all call execution dates. Checked / reported: 2026-10-20 · Samples: 400 Snapshot: 2026-10-08 · Revision: 8October2026; decideStudy.CSyRc3i6.js · Published summary observations; upstream task/result terms retained Result source · Benchmark source
Vals Mercury Decide study · Claim verification · all · estimated cost	0.00617 USD/1k cases	Setup & source Vals Mercury Decide study · Claim verification · all · estimated cost Vals8October2026 Mercury Decide direct Inception API; no version pin; one call/item, no malformed-response retries; same fixed instructions/labels as Jev study. Claims400SEC10-K items from48companies; LegalBench12preregistered subtasks×33questions. Historical$0.021/M input tariff; costs are usage×listedprice, not verified invoices. Routing threshold fit on30%, frozen before held-out70% (265items). Primary chart data module decideStudy.CSyRc3i6.js, parsed as literals without executing source code. Source decimals retained. ECE is a native fraction converted to percentage points; no mixture with Brier or ModelTested confidence score. Served checkpoint SHA/run dates unpublished; study publication8October is not all call execution dates. Primary chart data module decideStudy.CSyRc3i6.js, parsed as literals without executing source code. Source decimals retained. ECE is a native fraction converted to percentage points; no mixture with Brier or ModelTested confidence score. Served checkpoint SHA/run dates unpublished; study publication8October is not all call execution dates. Checked / reported: 2026-10-20 · Samples: 400 Snapshot: 2026-10-08 · Revision: 8October2026; decideStudy.CSyRc3i6.js · Published summary observations; upstream task/result terms retained Result source · Benchmark source
Vals Mercury Decide study · Claim verification · direct_support · accuracy	96.250%	Setup & source Vals Mercury Decide study · Claim verification · direct_support · accuracy Vals8October2026 Mercury Decide direct Inception API; no version pin; one call/item, no malformed-response retries; same fixed instructions/labels as Jev study. Claims400SEC10-K items from48companies; LegalBench12preregistered subtasks×33questions. Historical$0.021/M input tariff; costs are usage×listedprice, not verified invoices. Routing threshold fit on30%, frozen before held-out70% (265items). Primary chart data module decideStudy.CSyRc3i6.js, parsed as literals without executing source code. Source decimals retained. ECE is a native fraction converted to percentage points; no mixture with Brier or ModelTested confidence score. Served checkpoint SHA/run dates unpublished; study publication8October is not all call execution dates. Primary chart data module decideStudy.CSyRc3i6.js, parsed as literals without executing source code. Source decimals retained. ECE is a native fraction converted to percentage points; no mixture with Brier or ModelTested confidence score. Served checkpoint SHA/run dates unpublished; study publication8October is not all call execution dates. Checked / reported: 2026-10-20 · Samples: 80 Snapshot: 2026-10-08 · Revision: 8October2026; decideStudy.CSyRc3i6.js · Published summary observations; upstream task/result terms retained Result source · Benchmark source
Vals Mercury Decide study · Claim verification · direct_support · estimated cost	0.00608 USD/1k cases	Setup & source Vals Mercury Decide study · Claim verification · direct_support · estimated cost Vals8October2026 Mercury Decide direct Inception API; no version pin; one call/item, no malformed-response retries; same fixed instructions/labels as Jev study. Claims400SEC10-K items from48companies; LegalBench12preregistered subtasks×33questions. Historical$0.021/M input tariff; costs are usage×listedprice, not verified invoices. Routing threshold fit on30%, frozen before held-out70% (265items). Primary chart data module decideStudy.CSyRc3i6.js, parsed as literals without executing source code. Source decimals retained. ECE is a native fraction converted to percentage points; no mixture with Brier or ModelTested confidence score. Served checkpoint SHA/run dates unpublished; study publication8October is not all call execution dates. Primary chart data module decideStudy.CSyRc3i6.js, parsed as literals without executing source code. Source decimals retained. ECE is a native fraction converted to percentage points; no mixture with Brier or ModelTested confidence score. Served checkpoint SHA/run dates unpublished; study publication8October is not all call execution dates. Checked / reported: 2026-10-20 · Samples: 80 Snapshot: 2026-10-08 · Revision: 8October2026; decideStudy.CSyRc3i6.js · Published summary observations; upstream task/result terms retained Result source · Benchmark source
Vals Mercury Decide study · Claim verification · subtle_contradiction · accuracy	98.750%	Setup & source Vals Mercury Decide study · Claim verification · subtle_contradiction · accuracy Vals8October2026 Mercury Decide direct Inception API; no version pin; one call/item, no malformed-response retries; same fixed instructions/labels as Jev study. Claims400SEC10-K items from48companies; LegalBench12preregistered subtasks×33questions. Historical$0.021/M input tariff; costs are usage×listedprice, not verified invoices. Routing threshold fit on30%, frozen before held-out70% (265items). Primary chart data module decideStudy.CSyRc3i6.js, parsed as literals without executing source code. Source decimals retained. ECE is a native fraction converted to percentage points; no mixture with Brier or ModelTested confidence score. Served checkpoint SHA/run dates unpublished; study publication8October is not all call execution dates. Primary chart data module decideStudy.CSyRc3i6.js, parsed as literals without executing source code. Source decimals retained. ECE is a native fraction converted to percentage points; no mixture with Brier or ModelTested confidence score. Served checkpoint SHA/run dates unpublished; study publication8October is not all call execution dates. Checked / reported: 2026-10-20 · Samples: 80 Snapshot: 2026-10-08 · Revision: 8October2026; decideStudy.CSyRc3i6.js · Published summary observations; upstream task/result terms retained Result source · Benchmark source
Vals Mercury Decide study · Claim verification · subtle_contradiction · estimated cost	0.00613 USD/1k cases	Setup & source Vals Mercury Decide study · Claim verification · subtle_contradiction · estimated cost Vals8October2026 Mercury Decide direct Inception API; no version pin; one call/item, no malformed-response retries; same fixed instructions/labels as Jev study. Claims400SEC10-K items from48companies; LegalBench12preregistered subtasks×33questions. Historical$0.021/M input tariff; costs are usage×listedprice, not verified invoices. Routing threshold fit on30%, frozen before held-out70% (265items). Primary chart data module decideStudy.CSyRc3i6.js, parsed as literals without executing source code. Source decimals retained. ECE is a native fraction converted to percentage points; no mixture with Brier or ModelTested confidence score. Served checkpoint SHA/run dates unpublished; study publication8October is not all call execution dates. Primary chart data module decideStudy.CSyRc3i6.js, parsed as literals without executing source code. Source decimals retained. ECE is a native fraction converted to percentage points; no mixture with Brier or ModelTested confidence score. Served checkpoint SHA/run dates unpublished; study publication8October is not all call execution dates. Checked / reported: 2026-10-20 · Samples: 80 Snapshot: 2026-10-08 · Revision: 8October2026; decideStudy.CSyRc3i6.js · Published summary observations; upstream task/result terms retained Result source · Benchmark source
Vals Mercury Decide study · Claim verification · numeric_discrepancy · accuracy	100.000%	Setup & source Vals Mercury Decide study · Claim verification · numeric_discrepancy · accuracy Vals8October2026 Mercury Decide direct Inception API; no version pin; one call/item, no malformed-response retries; same fixed instructions/labels as Jev study. Claims400SEC10-K items from48companies; LegalBench12preregistered subtasks×33questions. Historical$0.021/M input tariff; costs are usage×listedprice, not verified invoices. Routing threshold fit on30%, frozen before held-out70% (265items). Primary chart data module decideStudy.CSyRc3i6.js, parsed as literals without executing source code. Source decimals retained. ECE is a native fraction converted to percentage points; no mixture with Brier or ModelTested confidence score. Served checkpoint SHA/run dates unpublished; study publication8October is not all call execution dates. Primary chart data module decideStudy.CSyRc3i6.js, parsed as literals without executing source code. Source decimals retained. ECE is a native fraction converted to percentage points; no mixture with Brier or ModelTested confidence score. Served checkpoint SHA/run dates unpublished; study publication8October is not all call execution dates. Checked / reported: 2026-10-20 · Samples: 80 Snapshot: 2026-10-08 · Revision: 8October2026; decideStudy.CSyRc3i6.js · Published summary observations; upstream task/result terms retained Result source · Benchmark source
Vals Mercury Decide study · Claim verification · numeric_discrepancy · estimated cost	0.00637 USD/1k cases	Setup & source Vals Mercury Decide study · Claim verification · numeric_discrepancy · estimated cost Vals8October2026 Mercury Decide direct Inception API; no version pin; one call/item, no malformed-response retries; same fixed instructions/labels as Jev study. Claims400SEC10-K items from48companies; LegalBench12preregistered subtasks×33questions. Historical$0.021/M input tariff; costs are usage×listedprice, not verified invoices. Routing threshold fit on30%, frozen before held-out70% (265items). Primary chart data module decideStudy.CSyRc3i6.js, parsed as literals without executing source code. Source decimals retained. ECE is a native fraction converted to percentage points; no mixture with Brier or ModelTested confidence score. Served checkpoint SHA/run dates unpublished; study publication8October is not all call execution dates. Primary chart data module decideStudy.CSyRc3i6.js, parsed as literals without executing source code. Source decimals retained. ECE is a native fraction converted to percentage points; no mixture with Brier or ModelTested confidence score. Served checkpoint SHA/run dates unpublished; study publication8October is not all call execution dates. Checked / reported: 2026-10-20 · Samples: 80 Snapshot: 2026-10-08 · Revision: 8October2026; decideStudy.CSyRc3i6.js · Published summary observations; upstream task/result terms retained Result source · Benchmark source
Vals Mercury Decide study · Claim verification · missing_evidence · accuracy	97.500%	Setup & source Vals Mercury Decide study · Claim verification · missing_evidence · accuracy Vals8October2026 Mercury Decide direct Inception API; no version pin; one call/item, no malformed-response retries; same fixed instructions/labels as Jev study. Claims400SEC10-K items from48companies; LegalBench12preregistered subtasks×33questions. Historical$0.021/M input tariff; costs are usage×listedprice, not verified invoices. Routing threshold fit on30%, frozen before held-out70% (265items). Primary chart data module decideStudy.CSyRc3i6.js, parsed as literals without executing source code. Source decimals retained. ECE is a native fraction converted to percentage points; no mixture with Brier or ModelTested confidence score. Served checkpoint SHA/run dates unpublished; study publication8October is not all call execution dates. Primary chart data module decideStudy.CSyRc3i6.js, parsed as literals without executing source code. Source decimals retained. ECE is a native fraction converted to percentage points; no mixture with Brier or ModelTested confidence score. Served checkpoint SHA/run dates unpublished; study publication8October is not all call execution dates. Checked / reported: 2026-10-20 · Samples: 80 Snapshot: 2026-10-08 · Revision: 8October2026; decideStudy.CSyRc3i6.js · Published summary observations; upstream task/result terms retained Result source · Benchmark source
Vals Mercury Decide study · Claim verification · missing_evidence · estimated cost	0.00557 USD/1k cases	Setup & source Vals Mercury Decide study · Claim verification · missing_evidence · estimated cost Vals8October2026 Mercury Decide direct Inception API; no version pin; one call/item, no malformed-response retries; same fixed instructions/labels as Jev study. Claims400SEC10-K items from48companies; LegalBench12preregistered subtasks×33questions. Historical$0.021/M input tariff; costs are usage×listedprice, not verified invoices. Routing threshold fit on30%, frozen before held-out70% (265items). Primary chart data module decideStudy.CSyRc3i6.js, parsed as literals without executing source code. Source decimals retained. ECE is a native fraction converted to percentage points; no mixture with Brier or ModelTested confidence score. Served checkpoint SHA/run dates unpublished; study publication8October is not all call execution dates. Primary chart data module decideStudy.CSyRc3i6.js, parsed as literals without executing source code. Source decimals retained. ECE is a native fraction converted to percentage points; no mixture with Brier or ModelTested confidence score. Served checkpoint SHA/run dates unpublished; study publication8October is not all call execution dates. Checked / reported: 2026-10-20 · Samples: 80 Snapshot: 2026-10-08 · Revision: 8October2026; decideStudy.CSyRc3i6.js · Published summary observations; upstream task/result terms retained Result source · Benchmark source
Vals Mercury Decide study · Claim verification · non_supporting_citation · accuracy	96.250%	Setup & source Vals Mercury Decide study · Claim verification · non_supporting_citation · accuracy Vals8October2026 Mercury Decide direct Inception API; no version pin; one call/item, no malformed-response retries; same fixed instructions/labels as Jev study. Claims400SEC10-K items from48companies; LegalBench12preregistered subtasks×33questions. Historical$0.021/M input tariff; costs are usage×listedprice, not verified invoices. Routing threshold fit on30%, frozen before held-out70% (265items). Primary chart data module decideStudy.CSyRc3i6.js, parsed as literals without executing source code. Source decimals retained. ECE is a native fraction converted to percentage points; no mixture with Brier or ModelTested confidence score. Served checkpoint SHA/run dates unpublished; study publication8October is not all call execution dates. Primary chart data module decideStudy.CSyRc3i6.js, parsed as literals without executing source code. Source decimals retained. ECE is a native fraction converted to percentage points; no mixture with Brier or ModelTested confidence score. Served checkpoint SHA/run dates unpublished; study publication8October is not all call execution dates. Checked / reported: 2026-10-20 · Samples: 80 Snapshot: 2026-10-08 · Revision: 8October2026; decideStudy.CSyRc3i6.js · Published summary observations; upstream task/result terms retained Result source · Benchmark source
Vals Mercury Decide study · Claim verification · non_supporting_citation · estimated cost	0.00670 USD/1k cases	Setup & source Vals Mercury Decide study · Claim verification · non_supporting_citation · estimated cost Vals8October2026 Mercury Decide direct Inception API; no version pin; one call/item, no malformed-response retries; same fixed instructions/labels as Jev study. Claims400SEC10-K items from48companies; LegalBench12preregistered subtasks×33questions. Historical$0.021/M input tariff; costs are usage×listedprice, not verified invoices. Routing threshold fit on30%, frozen before held-out70% (265items). Primary chart data module decideStudy.CSyRc3i6.js, parsed as literals without executing source code. Source decimals retained. ECE is a native fraction converted to percentage points; no mixture with Brier or ModelTested confidence score. Served checkpoint SHA/run dates unpublished; study publication8October is not all call execution dates. Primary chart data module decideStudy.CSyRc3i6.js, parsed as literals without executing source code. Source decimals retained. ECE is a native fraction converted to percentage points; no mixture with Brier or ModelTested confidence score. Served checkpoint SHA/run dates unpublished; study publication8October is not all call execution dates. Checked / reported: 2026-10-20 · Samples: 80 Snapshot: 2026-10-08 · Revision: 8October2026; decideStudy.CSyRc3i6.js · Published summary observations; upstream task/result terms retained Result source · Benchmark source
Vals Mercury Decide study · LegalBench33/task · international_citizenship_questions	69.697%	Setup & source Vals Mercury Decide study · LegalBench33/task · international_citizenship_questions Vals8October2026 Mercury Decide direct Inception API; no version pin; one call/item, no malformed-response retries; same fixed instructions/labels as Jev study. Claims400SEC10-K items from48companies; LegalBench12preregistered subtasks×33questions. Historical$0.021/M input tariff; costs are usage×listedprice, not verified invoices. Routing threshold fit on30%, frozen before held-out70% (265items). Primary chart data module decideStudy.CSyRc3i6.js, parsed as literals without executing source code. Source decimals retained. ECE is a native fraction converted to percentage points; no mixture with Brier or ModelTested confidence score. Served checkpoint SHA/run dates unpublished; study publication8October is not all call execution dates. Primary chart data module decideStudy.CSyRc3i6.js, parsed as literals without executing source code. Source decimals retained. ECE is a native fraction converted to percentage points; no mixture with Brier or ModelTested confidence score. Served checkpoint SHA/run dates unpublished; study publication8October is not all call execution dates. Checked / reported: 2026-10-20 · Samples: 33 Snapshot: 2026-10-08 · Revision: 8October2026; decideStudy.CSyRc3i6.js · Published summary observations; upstream task/result terms retained Result source · Benchmark source
Vals Mercury Decide study · LegalBench33/task · personal_jurisdiction	84.848%	Setup & source Vals Mercury Decide study · LegalBench33/task · personal_jurisdiction Vals8October2026 Mercury Decide direct Inception API; no version pin; one call/item, no malformed-response retries; same fixed instructions/labels as Jev study. Claims400SEC10-K items from48companies; LegalBench12preregistered subtasks×33questions. Historical$0.021/M input tariff; costs are usage×listedprice, not verified invoices. Routing threshold fit on30%, frozen before held-out70% (265items). Primary chart data module decideStudy.CSyRc3i6.js, parsed as literals without executing source code. Source decimals retained. ECE is a native fraction converted to percentage points; no mixture with Brier or ModelTested confidence score. Served checkpoint SHA/run dates unpublished; study publication8October is not all call execution dates. Primary chart data module decideStudy.CSyRc3i6.js, parsed as literals without executing source code. Source decimals retained. ECE is a native fraction converted to percentage points; no mixture with Brier or ModelTested confidence score. Served checkpoint SHA/run dates unpublished; study publication8October is not all call execution dates. Checked / reported: 2026-10-20 · Samples: 33 Snapshot: 2026-10-08 · Revision: 8October2026; decideStudy.CSyRc3i6.js · Published summary observations; upstream task/result terms retained Result source · Benchmark source
Vals Mercury Decide study · LegalBench33/task · learned_hands_consumer	75.758%	Setup & source Vals Mercury Decide study · LegalBench33/task · learned_hands_consumer Vals8October2026 Mercury Decide direct Inception API; no version pin; one call/item, no malformed-response retries; same fixed instructions/labels as Jev study. Claims400SEC10-K items from48companies; LegalBench12preregistered subtasks×33questions. Historical$0.021/M input tariff; costs are usage×listedprice, not verified invoices. Routing threshold fit on30%, frozen before held-out70% (265items). Primary chart data module decideStudy.CSyRc3i6.js, parsed as literals without executing source code. Source decimals retained. ECE is a native fraction converted to percentage points; no mixture with Brier or ModelTested confidence score. Served checkpoint SHA/run dates unpublished; study publication8October is not all call execution dates. Primary chart data module decideStudy.CSyRc3i6.js, parsed as literals without executing source code. Source decimals retained. ECE is a native fraction converted to percentage points; no mixture with Brier or ModelTested confidence score. Served checkpoint SHA/run dates unpublished; study publication8October is not all call execution dates. Checked / reported: 2026-10-20 · Samples: 33 Snapshot: 2026-10-08 · Revision: 8October2026; decideStudy.CSyRc3i6.js · Published summary observations; upstream task/result terms retained Result source · Benchmark source
Vals Mercury Decide study · LegalBench33/task · nys_judicial_ethics	78.788%	Setup & source Vals Mercury Decide study · LegalBench33/task · nys_judicial_ethics Vals8October2026 Mercury Decide direct Inception API; no version pin; one call/item, no malformed-response retries; same fixed instructions/labels as Jev study. Claims400SEC10-K items from48companies; LegalBench12preregistered subtasks×33questions. Historical$0.021/M input tariff; costs are usage×listedprice, not verified invoices. Routing threshold fit on30%, frozen before held-out70% (265items). Primary chart data module decideStudy.CSyRc3i6.js, parsed as literals without executing source code. Source decimals retained. ECE is a native fraction converted to percentage points; no mixture with Brier or ModelTested confidence score. Served checkpoint SHA/run dates unpublished; study publication8October is not all call execution dates. Primary chart data module decideStudy.CSyRc3i6.js, parsed as literals without executing source code. Source decimals retained. ECE is a native fraction converted to percentage points; no mixture with Brier or ModelTested confidence score. Served checkpoint SHA/run dates unpublished; study publication8October is not all call execution dates. Checked / reported: 2026-10-20 · Samples: 33 Snapshot: 2026-10-08 · Revision: 8October2026; decideStudy.CSyRc3i6.js · Published summary observations; upstream task/result terms retained Result source · Benchmark source
Vals Mercury Decide study · LegalBench33/task · abercrombie	81.818%	Setup & source Vals Mercury Decide study · LegalBench33/task · abercrombie Vals8October2026 Mercury Decide direct Inception API; no version pin; one call/item, no malformed-response retries; same fixed instructions/labels as Jev study. Claims400SEC10-K items from48companies; LegalBench12preregistered subtasks×33questions. Historical$0.021/M input tariff; costs are usage×listedprice, not verified invoices. Routing threshold fit on30%, frozen before held-out70% (265items). Primary chart data module decideStudy.CSyRc3i6.js, parsed as literals without executing source code. Source decimals retained. ECE is a native fraction converted to percentage points; no mixture with Brier or ModelTested confidence score. Served checkpoint SHA/run dates unpublished; study publication8October is not all call execution dates. Primary chart data module decideStudy.CSyRc3i6.js, parsed as literals without executing source code. Source decimals retained. ECE is a native fraction converted to percentage points; no mixture with Brier or ModelTested confidence score. Served checkpoint SHA/run dates unpublished; study publication8October is not all call execution dates. Checked / reported: 2026-10-20 · Samples: 33 Snapshot: 2026-10-08 · Revision: 8October2026; decideStudy.CSyRc3i6.js · Published summary observations; upstream task/result terms retained Result source · Benchmark source
Vals Mercury Decide study · LegalBench33/task · definition_classification	90.909%	Setup & source Vals Mercury Decide study · LegalBench33/task · definition_classification Vals8October2026 Mercury Decide direct Inception API; no version pin; one call/item, no malformed-response retries; same fixed instructions/labels as Jev study. Claims400SEC10-K items from48companies; LegalBench12preregistered subtasks×33questions. Historical$0.021/M input tariff; costs are usage×listedprice, not verified invoices. Routing threshold fit on30%, frozen before held-out70% (265items). Primary chart data module decideStudy.CSyRc3i6.js, parsed as literals without executing source code. Source decimals retained. ECE is a native fraction converted to percentage points; no mixture with Brier or ModelTested confidence score. Served checkpoint SHA/run dates unpublished; study publication8October is not all call execution dates. Primary chart data module decideStudy.CSyRc3i6.js, parsed as literals without executing source code. Source decimals retained. ECE is a native fraction converted to percentage points; no mixture with Brier or ModelTested confidence score. Served checkpoint SHA/run dates unpublished; study publication8October is not all call execution dates. Checked / reported: 2026-10-20 · Samples: 33 Snapshot: 2026-10-08 · Revision: 8October2026; decideStudy.CSyRc3i6.js · Published summary observations; upstream task/result terms retained Result source · Benchmark source
Vals Mercury Decide study · LegalBench33/task · hearsay	84.848%	Setup & source Vals Mercury Decide study · LegalBench33/task · hearsay Vals8October2026 Mercury Decide direct Inception API; no version pin; one call/item, no malformed-response retries; same fixed instructions/labels as Jev study. Claims400SEC10-K items from48companies; LegalBench12preregistered subtasks×33questions. Historical$0.021/M input tariff; costs are usage×listedprice, not verified invoices. Routing threshold fit on30%, frozen before held-out70% (265items). Primary chart data module decideStudy.CSyRc3i6.js, parsed as literals without executing source code. Source decimals retained. ECE is a native fraction converted to percentage points; no mixture with Brier or ModelTested confidence score. Served checkpoint SHA/run dates unpublished; study publication8October is not all call execution dates. Primary chart data module decideStudy.CSyRc3i6.js, parsed as literals without executing source code. Source decimals retained. ECE is a native fraction converted to percentage points; no mixture with Brier or ModelTested confidence score. Served checkpoint SHA/run dates unpublished; study publication8October is not all call execution dates. Checked / reported: 2026-10-20 · Samples: 33 Snapshot: 2026-10-08 · Revision: 8October2026; decideStudy.CSyRc3i6.js · Published summary observations; upstream task/result terms retained Result source · Benchmark source
Vals Mercury Decide study · LegalBench33/task · learned_hands_crime	87.879%	Setup & source Vals Mercury Decide study · LegalBench33/task · learned_hands_crime Vals8October2026 Mercury Decide direct Inception API; no version pin; one call/item, no malformed-response retries; same fixed instructions/labels as Jev study. Claims400SEC10-K items from48companies; LegalBench12preregistered subtasks×33questions. Historical$0.021/M input tariff; costs are usage×listedprice, not verified invoices. Routing threshold fit on30%, frozen before held-out70% (265items). Primary chart data module decideStudy.CSyRc3i6.js, parsed as literals without executing source code. Source decimals retained. ECE is a native fraction converted to percentage points; no mixture with Brier or ModelTested confidence score. Served checkpoint SHA/run dates unpublished; study publication8October is not all call execution dates. Primary chart data module decideStudy.CSyRc3i6.js, parsed as literals without executing source code. Source decimals retained. ECE is a native fraction converted to percentage points; no mixture with Brier or ModelTested confidence score. Served checkpoint SHA/run dates unpublished; study publication8October is not all call execution dates. Checked / reported: 2026-10-20 · Samples: 33 Snapshot: 2026-10-08 · Revision: 8October2026; decideStudy.CSyRc3i6.js · Published summary observations; upstream task/result terms retained Result source · Benchmark source
Vals Mercury Decide study · LegalBench33/task · overruling	93.939%	Setup & source Vals Mercury Decide study · LegalBench33/task · overruling Vals8October2026 Mercury Decide direct Inception API; no version pin; one call/item, no malformed-response retries; same fixed instructions/labels as Jev study. Claims400SEC10-K items from48companies; LegalBench12preregistered subtasks×33questions. Historical$0.021/M input tariff; costs are usage×listedprice, not verified invoices. Routing threshold fit on30%, frozen before held-out70% (265items). Primary chart data module decideStudy.CSyRc3i6.js, parsed as literals without executing source code. Source decimals retained. ECE is a native fraction converted to percentage points; no mixture with Brier or ModelTested confidence score. Served checkpoint SHA/run dates unpublished; study publication8October is not all call execution dates. Primary chart data module decideStudy.CSyRc3i6.js, parsed as literals without executing source code. Source decimals retained. ECE is a native fraction converted to percentage points; no mixture with Brier or ModelTested confidence score. Served checkpoint SHA/run dates unpublished; study publication8October is not all call execution dates. Checked / reported: 2026-10-20 · Samples: 33 Snapshot: 2026-10-08 · Revision: 8October2026; decideStudy.CSyRc3i6.js · Published summary observations; upstream task/result terms retained Result source · Benchmark source
Vals Mercury Decide study · LegalBench33/task · contract_nli_confidentiality_of_agreement	96.970%	Setup & source Vals Mercury Decide study · LegalBench33/task · contract_nli_confidentiality_of_agreement Vals8October2026 Mercury Decide direct Inception API; no version pin; one call/item, no malformed-response retries; same fixed instructions/labels as Jev study. Claims400SEC10-K items from48companies; LegalBench12preregistered subtasks×33questions. Historical$0.021/M input tariff; costs are usage×listedprice, not verified invoices. Routing threshold fit on30%, frozen before held-out70% (265items). Primary chart data module decideStudy.CSyRc3i6.js, parsed as literals without executing source code. Source decimals retained. ECE is a native fraction converted to percentage points; no mixture with Brier or ModelTested confidence score. Served checkpoint SHA/run dates unpublished; study publication8October is not all call execution dates. Primary chart data module decideStudy.CSyRc3i6.js, parsed as literals without executing source code. Source decimals retained. ECE is a native fraction converted to percentage points; no mixture with Brier or ModelTested confidence score. Served checkpoint SHA/run dates unpublished; study publication8October is not all call execution dates. Checked / reported: 2026-10-20 · Samples: 33 Snapshot: 2026-10-08 · Revision: 8October2026; decideStudy.CSyRc3i6.js · Published summary observations; upstream task/result terms retained Result source · Benchmark source
Vals Mercury Decide study · LegalBench33/task · contract_nli_permissible_copy	96.970%	Setup & source Vals Mercury Decide study · LegalBench33/task · contract_nli_permissible_copy Vals8October2026 Mercury Decide direct Inception API; no version pin; one call/item, no malformed-response retries; same fixed instructions/labels as Jev study. Claims400SEC10-K items from48companies; LegalBench12preregistered subtasks×33questions. Historical$0.021/M input tariff; costs are usage×listedprice, not verified invoices. Routing threshold fit on30%, frozen before held-out70% (265items). Primary chart data module decideStudy.CSyRc3i6.js, parsed as literals without executing source code. Source decimals retained. ECE is a native fraction converted to percentage points; no mixture with Brier or ModelTested confidence score. Served checkpoint SHA/run dates unpublished; study publication8October is not all call execution dates. Primary chart data module decideStudy.CSyRc3i6.js, parsed as literals without executing source code. Source decimals retained. ECE is a native fraction converted to percentage points; no mixture with Brier or ModelTested confidence score. Served checkpoint SHA/run dates unpublished; study publication8October is not all call execution dates. Checked / reported: 2026-10-20 · Samples: 33 Snapshot: 2026-10-08 · Revision: 8October2026; decideStudy.CSyRc3i6.js · Published summary observations; upstream task/result terms retained Result source · Benchmark source
Vals Mercury Decide study · LegalBench33/task · cuad_governing_law	100.000%	Setup & source Vals Mercury Decide study · LegalBench33/task · cuad_governing_law Vals8October2026 Mercury Decide direct Inception API; no version pin; one call/item, no malformed-response retries; same fixed instructions/labels as Jev study. Claims400SEC10-K items from48companies; LegalBench12preregistered subtasks×33questions. Historical$0.021/M input tariff; costs are usage×listedprice, not verified invoices. Routing threshold fit on30%, frozen before held-out70% (265items). Primary chart data module decideStudy.CSyRc3i6.js, parsed as literals without executing source code. Source decimals retained. ECE is a native fraction converted to percentage points; no mixture with Brier or ModelTested confidence score. Served checkpoint SHA/run dates unpublished; study publication8October is not all call execution dates. Primary chart data module decideStudy.CSyRc3i6.js, parsed as literals without executing source code. Source decimals retained. ECE is a native fraction converted to percentage points; no mixture with Brier or ModelTested confidence score. Served checkpoint SHA/run dates unpublished; study publication8October is not all call execution dates. Checked / reported: 2026-10-20 · Samples: 33 Snapshot: 2026-10-08 · Revision: 8October2026; decideStudy.CSyRc3i6.js · Published summary observations; upstream task/result terms retained Result source · Benchmark source
Vals Mercury Decide study · LegalBench396 · accuracy	86.900%	Setup & source Vals Mercury Decide study · LegalBench396 · accuracy Vals8October2026 Mercury Decide direct Inception API; no version pin; one call/item, no malformed-response retries; same fixed instructions/labels as Jev study. Claims400SEC10-K items from48companies; LegalBench12preregistered subtasks×33questions. Historical$0.021/M input tariff; costs are usage×listedprice, not verified invoices. Routing threshold fit on30%, frozen before held-out70% (265items). Primary chart data module decideStudy.CSyRc3i6.js, parsed as literals without executing source code. Source decimals retained. ECE is a native fraction converted to percentage points; no mixture with Brier or ModelTested confidence score. Served checkpoint SHA/run dates unpublished; study publication8October is not all call execution dates. Primary chart data module decideStudy.CSyRc3i6.js, parsed as literals without executing source code. Source decimals retained. ECE is a native fraction converted to percentage points; no mixture with Brier or ModelTested confidence score. Served checkpoint SHA/run dates unpublished; study publication8October is not all call execution dates. Checked / reported: 2026-10-20 · Samples: 396 Snapshot: 2026-10-08 · Revision: 8October2026; decideStudy.CSyRc3i6.js · Published summary observations; upstream task/result terms retained Result source · Benchmark source
Vals Mercury Decide study · LegalBench396 · class-balanced accuracy	82.100%	Setup & source Vals Mercury Decide study · LegalBench396 · class-balanced accuracy Vals8October2026 Mercury Decide direct Inception API; no version pin; one call/item, no malformed-response retries; same fixed instructions/labels as Jev study. Claims400SEC10-K items from48companies; LegalBench12preregistered subtasks×33questions. Historical$0.021/M input tariff; costs are usage×listedprice, not verified invoices. Routing threshold fit on30%, frozen before held-out70% (265items). Primary chart data module decideStudy.CSyRc3i6.js, parsed as literals without executing source code. Source decimals retained. ECE is a native fraction converted to percentage points; no mixture with Brier or ModelTested confidence score. Served checkpoint SHA/run dates unpublished; study publication8October is not all call execution dates. Primary chart data module decideStudy.CSyRc3i6.js, parsed as literals without executing source code. Source decimals retained. ECE is a native fraction converted to percentage points; no mixture with Brier or ModelTested confidence score. Served checkpoint SHA/run dates unpublished; study publication8October is not all call execution dates. Checked / reported: 2026-10-20 · Samples: 396 Snapshot: 2026-10-08 · Revision: 8October2026; decideStudy.CSyRc3i6.js · Published summary observations; upstream task/result terms retained Result source · Benchmark source
Vals Mercury Decide study · Claims · ECE	1.100 pp	Setup & source Vals Mercury Decide study · Claims · ECE Vals8October2026 Mercury Decide direct Inception API; no version pin; one call/item, no malformed-response retries; same fixed instructions/labels as Jev study. Claims400SEC10-K items from48companies; LegalBench12preregistered subtasks×33questions. Historical$0.021/M input tariff; costs are usage×listedprice, not verified invoices. Routing threshold fit on30%, frozen before held-out70% (265items). Primary chart data module decideStudy.CSyRc3i6.js, parsed as literals without executing source code. Source decimals retained. ECE is a native fraction converted to percentage points; no mixture with Brier or ModelTested confidence score. Served checkpoint SHA/run dates unpublished; study publication8October is not all call execution dates. Primary chart data module decideStudy.CSyRc3i6.js, parsed as literals without executing source code. Source decimals retained. ECE is a native fraction converted to percentage points; no mixture with Brier or ModelTested confidence score. Served checkpoint SHA/run dates unpublished; study publication8October is not all call execution dates. Checked / reported: 2026-10-20 · Samples: 400 Snapshot: 2026-10-08 · Revision: 8October2026; decideStudy.CSyRc3i6.js · Published summary observations; upstream task/result terms retained Result source · Benchmark source
Vals Mercury Decide study · LegalBench396 · ECE	5.800 pp	Setup & source Vals Mercury Decide study · LegalBench396 · ECE Vals8October2026 Mercury Decide direct Inception API; no version pin; one call/item, no malformed-response retries; same fixed instructions/labels as Jev study. Claims400SEC10-K items from48companies; LegalBench12preregistered subtasks×33questions. Historical$0.021/M input tariff; costs are usage×listedprice, not verified invoices. Routing threshold fit on30%, frozen before held-out70% (265items). Primary chart data module decideStudy.CSyRc3i6.js, parsed as literals without executing source code. Source decimals retained. ECE is a native fraction converted to percentage points; no mixture with Brier or ModelTested confidence score. Served checkpoint SHA/run dates unpublished; study publication8October is not all call execution dates. Primary chart data module decideStudy.CSyRc3i6.js, parsed as literals without executing source code. Source decimals retained. ECE is a native fraction converted to percentage points; no mixture with Brier or ModelTested confidence score. Served checkpoint SHA/run dates unpublished; study publication8October is not all call execution dates. Checked / reported: 2026-10-20 · Samples: 396 Snapshot: 2026-10-08 · Revision: 8October2026; decideStudy.CSyRc3i6.js · Published summary observations; upstream task/result terms retained Result source · Benchmark source
Vals Mercury Decide study · Held-out claims · routing coverage at1%budget	93.960%	Setup & source Vals Mercury Decide study · Held-out claims · routing coverage at1%budget Vals8October2026 Mercury Decide direct Inception API; no version pin; one call/item, no malformed-response retries; same fixed instructions/labels as Jev study. Claims400SEC10-K items from48companies; LegalBench12preregistered subtasks×33questions. Historical$0.021/M input tariff; costs are usage×listedprice, not verified invoices. Routing threshold fit on30%, frozen before held-out70% (265items). Primary chart data module decideStudy.CSyRc3i6.js, parsed as literals without executing source code. Source decimals retained. ECE is a native fraction converted to percentage points; no mixture with Brier or ModelTested confidence score. Served checkpoint SHA/run dates unpublished; study publication8October is not all call execution dates. Primary chart data module decideStudy.CSyRc3i6.js, parsed as literals without executing source code. Source decimals retained. ECE is a native fraction converted to percentage points; no mixture with Brier or ModelTested confidence score. Served checkpoint SHA/run dates unpublished; study publication8October is not all call execution dates. Checked / reported: 2026-10-20 · Samples: 265 Snapshot: 2026-10-08 · Revision: 8October2026; decideStudy.CSyRc3i6.js · Published summary observations; upstream task/result terms retained Result source · Benchmark source
Vals Mercury Decide study · Held-out claims · realized error at1%budget	0.803%	Setup & source Vals Mercury Decide study · Held-out claims · realized error at1%budget Vals8October2026 Mercury Decide direct Inception API; no version pin; one call/item, no malformed-response retries; same fixed instructions/labels as Jev study. Claims400SEC10-K items from48companies; LegalBench12preregistered subtasks×33questions. Historical$0.021/M input tariff; costs are usage×listedprice, not verified invoices. Routing threshold fit on30%, frozen before held-out70% (265items). Primary chart data module decideStudy.CSyRc3i6.js, parsed as literals without executing source code. Source decimals retained. ECE is a native fraction converted to percentage points; no mixture with Brier or ModelTested confidence score. Served checkpoint SHA/run dates unpublished; study publication8October is not all call execution dates. Primary chart data module decideStudy.CSyRc3i6.js, parsed as literals without executing source code. Source decimals retained. ECE is a native fraction converted to percentage points; no mixture with Brier or ModelTested confidence score. Served checkpoint SHA/run dates unpublished; study publication8October is not all call execution dates. Checked / reported: 2026-10-20 · Samples: 265 Snapshot: 2026-10-08 · Revision: 8October2026; decideStudy.CSyRc3i6.js · Published summary observations; upstream task/result terms retained Result source · Benchmark source
Vals Mercury Decide study · Long document · 1judgments · median	0.1590 s	Setup & source Vals Mercury Decide study · Long document · 1judgments · median Vals8October2026 Mercury Decide direct Inception API; no version pin; one call/item, no malformed-response retries; same fixed instructions/labels as Jev study. Claims400SEC10-K items from48companies; LegalBench12preregistered subtasks×33questions. Historical$0.021/M input tariff; costs are usage×listedprice, not verified invoices. Routing threshold fit on30%, frozen before held-out70% (265items). Primary chart data module decideStudy.CSyRc3i6.js, parsed as literals without executing source code. Source decimals retained. ECE is a native fraction converted to percentage points; no mixture with Brier or ModelTested confidence score. Served checkpoint SHA/run dates unpublished; study publication8October is not all call execution dates. Primary chart data module decideStudy.CSyRc3i6.js, parsed as literals without executing source code. Source decimals retained. ECE is a native fraction converted to percentage points; no mixture with Brier or ModelTested confidence score. Served checkpoint SHA/run dates unpublished; study publication8October is not all call execution dates. Checked / reported: 2026-10-20 · Samples: Not published Snapshot: 2026-10-08 · Revision: 8October2026; decideStudy.CSyRc3i6.js · Published summary observations; upstream task/result terms retained Result source · Benchmark source
Vals Mercury Decide study · Long document · 8judgments · median	0.3068 s	Setup & source Vals Mercury Decide study · Long document · 8judgments · median Vals8October2026 Mercury Decide direct Inception API; no version pin; one call/item, no malformed-response retries; same fixed instructions/labels as Jev study. Claims400SEC10-K items from48companies; LegalBench12preregistered subtasks×33questions. Historical$0.021/M input tariff; costs are usage×listedprice, not verified invoices. Routing threshold fit on30%, frozen before held-out70% (265items). Primary chart data module decideStudy.CSyRc3i6.js, parsed as literals without executing source code. Source decimals retained. ECE is a native fraction converted to percentage points; no mixture with Brier or ModelTested confidence score. Served checkpoint SHA/run dates unpublished; study publication8October is not all call execution dates. Primary chart data module decideStudy.CSyRc3i6.js, parsed as literals without executing source code. Source decimals retained. ECE is a native fraction converted to percentage points; no mixture with Brier or ModelTested confidence score. Served checkpoint SHA/run dates unpublished; study publication8October is not all call execution dates. Checked / reported: 2026-10-20 · Samples: Not published Snapshot: 2026-10-08 · Revision: 8October2026; decideStudy.CSyRc3i6.js · Published summary observations; upstream task/result terms retained Result source · Benchmark source
Vals Mercury Decide study · Long document · 32judgments · median	0.3774 s	Setup & source Vals Mercury Decide study · Long document · 32judgments · median Vals8October2026 Mercury Decide direct Inception API; no version pin; one call/item, no malformed-response retries; same fixed instructions/labels as Jev study. Claims400SEC10-K items from48companies; LegalBench12preregistered subtasks×33questions. Historical$0.021/M input tariff; costs are usage×listedprice, not verified invoices. Routing threshold fit on30%, frozen before held-out70% (265items). Primary chart data module decideStudy.CSyRc3i6.js, parsed as literals without executing source code. Source decimals retained. ECE is a native fraction converted to percentage points; no mixture with Brier or ModelTested confidence score. Served checkpoint SHA/run dates unpublished; study publication8October is not all call execution dates. Primary chart data module decideStudy.CSyRc3i6.js, parsed as literals without executing source code. Source decimals retained. ECE is a native fraction converted to percentage points; no mixture with Brier or ModelTested confidence score. Served checkpoint SHA/run dates unpublished; study publication8October is not all call execution dates. Checked / reported: 2026-10-20 · Samples: Not published Snapshot: 2026-10-08 · Revision: 8October2026; decideStudy.CSyRc3i6.js · Published summary observations; upstream task/result terms retained Result source · Benchmark source
Vals Mercury Decide study · Long document · 64judgments · median	0.3992 s	Setup & source Vals Mercury Decide study · Long document · 64judgments · median Vals8October2026 Mercury Decide direct Inception API; no version pin; one call/item, no malformed-response retries; same fixed instructions/labels as Jev study. Claims400SEC10-K items from48companies; LegalBench12preregistered subtasks×33questions. Historical$0.021/M input tariff; costs are usage×listedprice, not verified invoices. Routing threshold fit on30%, frozen before held-out70% (265items). Primary chart data module decideStudy.CSyRc3i6.js, parsed as literals without executing source code. Source decimals retained. ECE is a native fraction converted to percentage points; no mixture with Brier or ModelTested confidence score. Served checkpoint SHA/run dates unpublished; study publication8October is not all call execution dates. Primary chart data module decideStudy.CSyRc3i6.js, parsed as literals without executing source code. Source decimals retained. ECE is a native fraction converted to percentage points; no mixture with Brier or ModelTested confidence score. Served checkpoint SHA/run dates unpublished; study publication8October is not all call execution dates. Checked / reported: 2026-10-20 · Samples: Not published Snapshot: 2026-10-08 · Revision: 8October2026; decideStudy.CSyRc3i6.js · Published summary observations; upstream task/result terms retained Result source · Benchmark source
Vals Mercury Decide study · Paired claim latency / Jev	1.440 ratio	Setup & source Vals Mercury Decide study · Paired claim latency / Jev Vals8October2026 Mercury Decide direct Inception API; no version pin; one call/item, no malformed-response retries; same fixed instructions/labels as Jev study. Claims400SEC10-K items from48companies; LegalBench12preregistered subtasks×33questions. Historical$0.021/M input tariff; costs are usage×listedprice, not verified invoices. Routing threshold fit on30%, frozen before held-out70% (265items). Primary chart data module decideStudy.CSyRc3i6.js, parsed as literals without executing source code. Source decimals retained. ECE is a native fraction converted to percentage points; no mixture with Brier or ModelTested confidence score. Served checkpoint SHA/run dates unpublished; study publication8October is not all call execution dates. Primary chart data module decideStudy.CSyRc3i6.js, parsed as literals without executing source code. Source decimals retained. ECE is a native fraction converted to percentage points; no mixture with Brier or ModelTested confidence score. Served checkpoint SHA/run dates unpublished; study publication8October is not all call execution dates. Checked / reported: 2026-10-20 · Samples: Not published Snapshot: 2026-10-08 · Revision: 8October2026; decideStudy.CSyRc3i6.js · Published summary observations; upstream task/result terms retained Result source · Benchmark source
Vals Mercury Decide study · Long document64judgments · estimated cost	0.00220 USD/call	Setup & source Vals Mercury Decide study · Long document64judgments · estimated cost Vals8October2026 Mercury Decide direct Inception API; no version pin; one call/item, no malformed-response retries; same fixed instructions/labels as Jev study. Claims400SEC10-K items from48companies; LegalBench12preregistered subtasks×33questions. Historical$0.021/M input tariff; costs are usage×listedprice, not verified invoices. Routing threshold fit on30%, frozen before held-out70% (265items). Primary chart data module decideStudy.CSyRc3i6.js, parsed as literals without executing source code. Source decimals retained. ECE is a native fraction converted to percentage points; no mixture with Brier or ModelTested confidence score. Served checkpoint SHA/run dates unpublished; study publication8October is not all call execution dates. Primary chart data module decideStudy.CSyRc3i6.js, parsed as literals without executing source code. Source decimals retained. ECE is a native fraction converted to percentage points; no mixture with Brier or ModelTested confidence score. Served checkpoint SHA/run dates unpublished; study publication8October is not all call execution dates. Checked / reported: 2026-10-20 · Samples: Not published Snapshot: 2026-10-08 · Revision: 8October2026; decideStudy.CSyRc3i6.js · Published summary observations; upstream task/result terms retained Result source · Benchmark source

Mercury Decide · Evaluator-published · Benchmark Heaven
Evaluation	Result	Evidence
Benchmark Heaven JevBench1.6.1 · calibration axis	81.343	Setup & source Benchmark Heaven JevBench1.6.1 · calibration axis Benchmark Heaven v1.6.1; measured_in=v1.6.1; named hosted inception/mercury-decide:free.1500decisions (300public+1200sealed); source identity {"model_pin": "provider-reported measurement model slug: inception/mercury-decide:free", "last_measured_on": "2026-10-07", "endpoint_condition": "measured 2026-10-07 on the live v1.6.0/v1.6.1 pool (inputs/input-selfhosted-1500.jsonl, sha256 901983ae...), COMPLETE: 1,500 of 1,500 dispatched, 1,477 answered, 23 invalid. Measured through the release lane's reviewed run_api_full_v161.py UNCHANGED, on Florian's 5 Oct amendment (hosted systems answer the same full S u P as self-hosted ones); provider inception, proxy openrouter, every sealed item exposure-logged before the first request. Florian work order of 6 Oct 2026 ('put it on JevBench', under https://x.com/airesearch12/status/2107569862486659111). NO MAPPING DECISION OF OURS: Mercury Decide publishes the same /v1/systemone contract as Jev and all three primitives come back in the canonical shape (noul a float; choice a label plus probabilities keyed by the item's own option labels; score probabilities keyed by level index '0'..'n-1' - 0-based, unlike ClassOne's), so the vendored typesafe adapter's request body, answer mapping and validations are reused UNCHANGED. or_decisions_r58.OpenRouterDecisionsAdapter rewrites ONLY the URL, because OpenRouter serves the contract at POST /api/alpha/decisions rather than /v1/systemone - its own 400 from chat/completions names that path - and it does so by calling the parent run(), so none of the reviewed mapping is duplicated. The pinned harness/vendor tree was NOT edited; the module is reached by PYTHONPATH. THE 23 LONG ITEMS: on this pool exactly 23 of 1,500 items are 77,857-82,318 tokens and the 24th largest is below 2,048 (run 57's own per-item counts on the identical pool, receipt LONGITEM-TOKENS-R58.json). They are also the 23 items the v1.6.1 cost rule excludes. THIS ROW REFUSES THEM: the context window is 32,768 tokens and all 23 came back as HTTP 422 with the provider's own 'upstream_error: Decision service could not complete the request' - a disclosed refusal, which the harness counts as the system's own rather than as an outage. The invalid set is EXACTLY those 23 and nothing else failed in 1,477 answers. RUN IN TWO PASSES BECAUSE OF A FREE-TIER BURST LIMIT, AND IT DOES NOT MOVE THE SPEED FIGURE: items 1-736 ran back to back at ~1.7 req/s until OpenRouter returned a header-less 429 ('rate_limit_exceeded: Request or spending limit exceeded'), which is run_v16's own stop rule; the row was resumed at 1 req/s and completed. The two passes' medians differ by 0.038 s (0.3624 s vs 0.3246 s over 1,477 answers; 0.3624 vs 0.3104 on the 124 items the Speed axis actually uses), and run_v16 sleeps the spacing BEFORE starting the timer, so the pacing is not in the measurement. Receipt LATENCY-SEGMENTS-mercury-R58.txt. Production API, so NO latency adjustment. Serial, one decision per request, no retry, no batching. add-requests run 58", "adapter_id": null, "measurement_date_status": "run output completion day (UTC)"} Distinct from Leanmcp JevBench and community HF Decision Index. WebMCP feed reportsv1.6.29 while page shell saysboardv1.7.46; preserve exact feed revision and source artifact. Intelligence is chance-corrected/weighted. Calibration is a mixed source index, not ECE. Cost and four-axis composite incorporate an explicitly labeled Mercury2.5 proxy tariff because the original7October run was free; this is historical source methodology, not current Mercury Decide pricing. No paid tariff is inferred. Source-computed values are retained; no Examenos recomputation. Per-type support/validity details are in each record. Distinct from Leanmcp JevBench and community HF Decision Index. WebMCP feed reportsv1.6.29 while page shell saysboardv1.7.46; preserve exact feed revision and source artifact. Intelligence is chance-corrected/weighted. Calibration is a mixed source index, not ECE. Cost and four-axis composite incorporate an explicitly labeled Mercury2.5 proxy tariff because the original7October run was free; this is historical source methodology, not current Mercury Decide pricing. No paid tariff is inferred. Source-computed values are retained; no Examenos recomputation. Per-type support/validity details are in each record. Checked / reported: 2026-10-20 · Samples: 1500 Snapshot: 2026-10-20 · Revision: v1.6.1; source_sha256=e15347783b8bedeeddd9017a831608a957f5bf0386f05902c50161ce0468f911 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
Benchmark Heaven JevBench1.6.1 · cost axis (historical proxy estimate)	62.083	Setup & source Benchmark Heaven JevBench1.6.1 · cost axis (historical proxy estimate) Benchmark Heaven v1.6.1; measured_in=v1.6.1; named hosted inception/mercury-decide:free.1500decisions (300public+1200sealed); source identity {"model_pin": "provider-reported measurement model slug: inception/mercury-decide:free", "last_measured_on": "2026-10-07", "endpoint_condition": "measured 2026-10-07 on the live v1.6.0/v1.6.1 pool (inputs/input-selfhosted-1500.jsonl, sha256 901983ae...), COMPLETE: 1,500 of 1,500 dispatched, 1,477 answered, 23 invalid. Measured through the release lane's reviewed run_api_full_v161.py UNCHANGED, on Florian's 5 Oct amendment (hosted systems answer the same full S u P as self-hosted ones); provider inception, proxy openrouter, every sealed item exposure-logged before the first request. Florian work order of 6 Oct 2026 ('put it on JevBench', under https://x.com/airesearch12/status/2107569862486659111). NO MAPPING DECISION OF OURS: Mercury Decide publishes the same /v1/systemone contract as Jev and all three primitives come back in the canonical shape (noul a float; choice a label plus probabilities keyed by the item's own option labels; score probabilities keyed by level index '0'..'n-1' - 0-based, unlike ClassOne's), so the vendored typesafe adapter's request body, answer mapping and validations are reused UNCHANGED. or_decisions_r58.OpenRouterDecisionsAdapter rewrites ONLY the URL, because OpenRouter serves the contract at POST /api/alpha/decisions rather than /v1/systemone - its own 400 from chat/completions names that path - and it does so by calling the parent run(), so none of the reviewed mapping is duplicated. The pinned harness/vendor tree was NOT edited; the module is reached by PYTHONPATH. THE 23 LONG ITEMS: on this pool exactly 23 of 1,500 items are 77,857-82,318 tokens and the 24th largest is below 2,048 (run 57's own per-item counts on the identical pool, receipt LONGITEM-TOKENS-R58.json). They are also the 23 items the v1.6.1 cost rule excludes. THIS ROW REFUSES THEM: the context window is 32,768 tokens and all 23 came back as HTTP 422 with the provider's own 'upstream_error: Decision service could not complete the request' - a disclosed refusal, which the harness counts as the system's own rather than as an outage. The invalid set is EXACTLY those 23 and nothing else failed in 1,477 answers. RUN IN TWO PASSES BECAUSE OF A FREE-TIER BURST LIMIT, AND IT DOES NOT MOVE THE SPEED FIGURE: items 1-736 ran back to back at ~1.7 req/s until OpenRouter returned a header-less 429 ('rate_limit_exceeded: Request or spending limit exceeded'), which is run_v16's own stop rule; the row was resumed at 1 req/s and completed. The two passes' medians differ by 0.038 s (0.3624 s vs 0.3246 s over 1,477 answers; 0.3624 vs 0.3104 on the 124 items the Speed axis actually uses), and run_v16 sleeps the spacing BEFORE starting the timer, so the pacing is not in the measurement. Receipt LATENCY-SEGMENTS-mercury-R58.txt. Production API, so NO latency adjustment. Serial, one decision per request, no retry, no batching. add-requests run 58", "adapter_id": null, "measurement_date_status": "run output completion day (UTC)"} Distinct from Leanmcp JevBench and community HF Decision Index. WebMCP feed reportsv1.6.29 while page shell saysboardv1.7.46; preserve exact feed revision and source artifact. Intelligence is chance-corrected/weighted. Calibration is a mixed source index, not ECE. Cost and four-axis composite incorporate an explicitly labeled Mercury2.5 proxy tariff because the original7October run was free; this is historical source methodology, not current Mercury Decide pricing. No paid tariff is inferred. Source-computed values are retained; no Examenos recomputation. Per-type support/validity details are in each record. Distinct from Leanmcp JevBench and community HF Decision Index. WebMCP feed reportsv1.6.29 while page shell saysboardv1.7.46; preserve exact feed revision and source artifact. Intelligence is chance-corrected/weighted. Calibration is a mixed source index, not ECE. Cost and four-axis composite incorporate an explicitly labeled Mercury2.5 proxy tariff because the original7October run was free; this is historical source methodology, not current Mercury Decide pricing. No paid tariff is inferred. Source-computed values are retained; no Examenos recomputation. Per-type support/validity details are in each record. Checked / reported: 2026-10-20 · Samples: 1500 Snapshot: 2026-10-20 · Revision: v1.6.1; source_sha256=e15347783b8bedeeddd9017a831608a957f5bf0386f05902c50161ce0468f911 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
Benchmark Heaven JevBench1.6.1 · intelligence axis	65.907	Setup & source Benchmark Heaven JevBench1.6.1 · intelligence axis Benchmark Heaven v1.6.1; measured_in=v1.6.1; named hosted inception/mercury-decide:free.1500decisions (300public+1200sealed); source identity {"model_pin": "provider-reported measurement model slug: inception/mercury-decide:free", "last_measured_on": "2026-10-07", "endpoint_condition": "measured 2026-10-07 on the live v1.6.0/v1.6.1 pool (inputs/input-selfhosted-1500.jsonl, sha256 901983ae...), COMPLETE: 1,500 of 1,500 dispatched, 1,477 answered, 23 invalid. Measured through the release lane's reviewed run_api_full_v161.py UNCHANGED, on Florian's 5 Oct amendment (hosted systems answer the same full S u P as self-hosted ones); provider inception, proxy openrouter, every sealed item exposure-logged before the first request. Florian work order of 6 Oct 2026 ('put it on JevBench', under https://x.com/airesearch12/status/2107569862486659111). NO MAPPING DECISION OF OURS: Mercury Decide publishes the same /v1/systemone contract as Jev and all three primitives come back in the canonical shape (noul a float; choice a label plus probabilities keyed by the item's own option labels; score probabilities keyed by level index '0'..'n-1' - 0-based, unlike ClassOne's), so the vendored typesafe adapter's request body, answer mapping and validations are reused UNCHANGED. or_decisions_r58.OpenRouterDecisionsAdapter rewrites ONLY the URL, because OpenRouter serves the contract at POST /api/alpha/decisions rather than /v1/systemone - its own 400 from chat/completions names that path - and it does so by calling the parent run(), so none of the reviewed mapping is duplicated. The pinned harness/vendor tree was NOT edited; the module is reached by PYTHONPATH. THE 23 LONG ITEMS: on this pool exactly 23 of 1,500 items are 77,857-82,318 tokens and the 24th largest is below 2,048 (run 57's own per-item counts on the identical pool, receipt LONGITEM-TOKENS-R58.json). They are also the 23 items the v1.6.1 cost rule excludes. THIS ROW REFUSES THEM: the context window is 32,768 tokens and all 23 came back as HTTP 422 with the provider's own 'upstream_error: Decision service could not complete the request' - a disclosed refusal, which the harness counts as the system's own rather than as an outage. The invalid set is EXACTLY those 23 and nothing else failed in 1,477 answers. RUN IN TWO PASSES BECAUSE OF A FREE-TIER BURST LIMIT, AND IT DOES NOT MOVE THE SPEED FIGURE: items 1-736 ran back to back at ~1.7 req/s until OpenRouter returned a header-less 429 ('rate_limit_exceeded: Request or spending limit exceeded'), which is run_v16's own stop rule; the row was resumed at 1 req/s and completed. The two passes' medians differ by 0.038 s (0.3624 s vs 0.3246 s over 1,477 answers; 0.3624 vs 0.3104 on the 124 items the Speed axis actually uses), and run_v16 sleeps the spacing BEFORE starting the timer, so the pacing is not in the measurement. Receipt LATENCY-SEGMENTS-mercury-R58.txt. Production API, so NO latency adjustment. Serial, one decision per request, no retry, no batching. add-requests run 58", "adapter_id": null, "measurement_date_status": "run output completion day (UTC)"} Distinct from Leanmcp JevBench and community HF Decision Index. WebMCP feed reportsv1.6.29 while page shell saysboardv1.7.46; preserve exact feed revision and source artifact. Intelligence is chance-corrected/weighted. Calibration is a mixed source index, not ECE. Cost and four-axis composite incorporate an explicitly labeled Mercury2.5 proxy tariff because the original7October run was free; this is historical source methodology, not current Mercury Decide pricing. No paid tariff is inferred. Source-computed values are retained; no Examenos recomputation. Per-type support/validity details are in each record. Distinct from Leanmcp JevBench and community HF Decision Index. WebMCP feed reportsv1.6.29 while page shell saysboardv1.7.46; preserve exact feed revision and source artifact. Intelligence is chance-corrected/weighted. Calibration is a mixed source index, not ECE. Cost and four-axis composite incorporate an explicitly labeled Mercury2.5 proxy tariff because the original7October run was free; this is historical source methodology, not current Mercury Decide pricing. No paid tariff is inferred. Source-computed values are retained; no Examenos recomputation. Per-type support/validity details are in each record. Checked / reported: 2026-10-20 · Samples: 1500 Snapshot: 2026-10-20 · Revision: v1.6.1; source_sha256=e15347783b8bedeeddd9017a831608a957f5bf0386f05902c50161ce0468f911 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
Benchmark Heaven JevBench1.6.1 · speed axis	85.478	Setup & source Benchmark Heaven JevBench1.6.1 · speed axis Benchmark Heaven v1.6.1; measured_in=v1.6.1; named hosted inception/mercury-decide:free.1500decisions (300public+1200sealed); source identity {"model_pin": "provider-reported measurement model slug: inception/mercury-decide:free", "last_measured_on": "2026-10-07", "endpoint_condition": "measured 2026-10-07 on the live v1.6.0/v1.6.1 pool (inputs/input-selfhosted-1500.jsonl, sha256 901983ae...), COMPLETE: 1,500 of 1,500 dispatched, 1,477 answered, 23 invalid. Measured through the release lane's reviewed run_api_full_v161.py UNCHANGED, on Florian's 5 Oct amendment (hosted systems answer the same full S u P as self-hosted ones); provider inception, proxy openrouter, every sealed item exposure-logged before the first request. Florian work order of 6 Oct 2026 ('put it on JevBench', under https://x.com/airesearch12/status/2107569862486659111). NO MAPPING DECISION OF OURS: Mercury Decide publishes the same /v1/systemone contract as Jev and all three primitives come back in the canonical shape (noul a float; choice a label plus probabilities keyed by the item's own option labels; score probabilities keyed by level index '0'..'n-1' - 0-based, unlike ClassOne's), so the vendored typesafe adapter's request body, answer mapping and validations are reused UNCHANGED. or_decisions_r58.OpenRouterDecisionsAdapter rewrites ONLY the URL, because OpenRouter serves the contract at POST /api/alpha/decisions rather than /v1/systemone - its own 400 from chat/completions names that path - and it does so by calling the parent run(), so none of the reviewed mapping is duplicated. The pinned harness/vendor tree was NOT edited; the module is reached by PYTHONPATH. THE 23 LONG ITEMS: on this pool exactly 23 of 1,500 items are 77,857-82,318 tokens and the 24th largest is below 2,048 (run 57's own per-item counts on the identical pool, receipt LONGITEM-TOKENS-R58.json). They are also the 23 items the v1.6.1 cost rule excludes. THIS ROW REFUSES THEM: the context window is 32,768 tokens and all 23 came back as HTTP 422 with the provider's own 'upstream_error: Decision service could not complete the request' - a disclosed refusal, which the harness counts as the system's own rather than as an outage. The invalid set is EXACTLY those 23 and nothing else failed in 1,477 answers. RUN IN TWO PASSES BECAUSE OF A FREE-TIER BURST LIMIT, AND IT DOES NOT MOVE THE SPEED FIGURE: items 1-736 ran back to back at ~1.7 req/s until OpenRouter returned a header-less 429 ('rate_limit_exceeded: Request or spending limit exceeded'), which is run_v16's own stop rule; the row was resumed at 1 req/s and completed. The two passes' medians differ by 0.038 s (0.3624 s vs 0.3246 s over 1,477 answers; 0.3624 vs 0.3104 on the 124 items the Speed axis actually uses), and run_v16 sleeps the spacing BEFORE starting the timer, so the pacing is not in the measurement. Receipt LATENCY-SEGMENTS-mercury-R58.txt. Production API, so NO latency adjustment. Serial, one decision per request, no retry, no batching. add-requests run 58", "adapter_id": null, "measurement_date_status": "run output completion day (UTC)"} Distinct from Leanmcp JevBench and community HF Decision Index. WebMCP feed reportsv1.6.29 while page shell saysboardv1.7.46; preserve exact feed revision and source artifact. Intelligence is chance-corrected/weighted. Calibration is a mixed source index, not ECE. Cost and four-axis composite incorporate an explicitly labeled Mercury2.5 proxy tariff because the original7October run was free; this is historical source methodology, not current Mercury Decide pricing. No paid tariff is inferred. Source-computed values are retained; no Examenos recomputation. Per-type support/validity details are in each record. Distinct from Leanmcp JevBench and community HF Decision Index. WebMCP feed reportsv1.6.29 while page shell saysboardv1.7.46; preserve exact feed revision and source artifact. Intelligence is chance-corrected/weighted. Calibration is a mixed source index, not ECE. Cost and four-axis composite incorporate an explicitly labeled Mercury2.5 proxy tariff because the original7October run was free; this is historical source methodology, not current Mercury Decide pricing. No paid tariff is inferred. Source-computed values are retained; no Examenos recomputation. Per-type support/validity details are in each record. Checked / reported: 2026-10-20 · Samples: 1500 Snapshot: 2026-10-20 · Revision: v1.6.1; source_sha256=e15347783b8bedeeddd9017a831608a957f5bf0386f05902c50161ce0468f911 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
Benchmark Heaven JevBench1.6.1 · Capability	73.625	Setup & source Benchmark Heaven JevBench1.6.1 · Capability Benchmark Heaven v1.6.1; measured_in=v1.6.1; named hosted inception/mercury-decide:free.1500decisions (300public+1200sealed); source identity {"model_pin": "provider-reported measurement model slug: inception/mercury-decide:free", "last_measured_on": "2026-10-07", "endpoint_condition": "measured 2026-10-07 on the live v1.6.0/v1.6.1 pool (inputs/input-selfhosted-1500.jsonl, sha256 901983ae...), COMPLETE: 1,500 of 1,500 dispatched, 1,477 answered, 23 invalid. Measured through the release lane's reviewed run_api_full_v161.py UNCHANGED, on Florian's 5 Oct amendment (hosted systems answer the same full S u P as self-hosted ones); provider inception, proxy openrouter, every sealed item exposure-logged before the first request. Florian work order of 6 Oct 2026 ('put it on JevBench', under https://x.com/airesearch12/status/2107569862486659111). NO MAPPING DECISION OF OURS: Mercury Decide publishes the same /v1/systemone contract as Jev and all three primitives come back in the canonical shape (noul a float; choice a label plus probabilities keyed by the item's own option labels; score probabilities keyed by level index '0'..'n-1' - 0-based, unlike ClassOne's), so the vendored typesafe adapter's request body, answer mapping and validations are reused UNCHANGED. or_decisions_r58.OpenRouterDecisionsAdapter rewrites ONLY the URL, because OpenRouter serves the contract at POST /api/alpha/decisions rather than /v1/systemone - its own 400 from chat/completions names that path - and it does so by calling the parent run(), so none of the reviewed mapping is duplicated. The pinned harness/vendor tree was NOT edited; the module is reached by PYTHONPATH. THE 23 LONG ITEMS: on this pool exactly 23 of 1,500 items are 77,857-82,318 tokens and the 24th largest is below 2,048 (run 57's own per-item counts on the identical pool, receipt LONGITEM-TOKENS-R58.json). They are also the 23 items the v1.6.1 cost rule excludes. THIS ROW REFUSES THEM: the context window is 32,768 tokens and all 23 came back as HTTP 422 with the provider's own 'upstream_error: Decision service could not complete the request' - a disclosed refusal, which the harness counts as the system's own rather than as an outage. The invalid set is EXACTLY those 23 and nothing else failed in 1,477 answers. RUN IN TWO PASSES BECAUSE OF A FREE-TIER BURST LIMIT, AND IT DOES NOT MOVE THE SPEED FIGURE: items 1-736 ran back to back at ~1.7 req/s until OpenRouter returned a header-less 429 ('rate_limit_exceeded: Request or spending limit exceeded'), which is run_v16's own stop rule; the row was resumed at 1 req/s and completed. The two passes' medians differ by 0.038 s (0.3624 s vs 0.3246 s over 1,477 answers; 0.3624 vs 0.3104 on the 124 items the Speed axis actually uses), and run_v16 sleeps the spacing BEFORE starting the timer, so the pacing is not in the measurement. Receipt LATENCY-SEGMENTS-mercury-R58.txt. Production API, so NO latency adjustment. Serial, one decision per request, no retry, no batching. add-requests run 58", "adapter_id": null, "measurement_date_status": "run output completion day (UTC)"} Distinct from Leanmcp JevBench and community HF Decision Index. WebMCP feed reportsv1.6.29 while page shell saysboardv1.7.46; preserve exact feed revision and source artifact. Intelligence is chance-corrected/weighted. Calibration is a mixed source index, not ECE. Cost and four-axis composite incorporate an explicitly labeled Mercury2.5 proxy tariff because the original7October run was free; this is historical source methodology, not current Mercury Decide pricing. No paid tariff is inferred. Source-computed values are retained; no Examenos recomputation. Per-type support/validity details are in each record. Distinct from Leanmcp JevBench and community HF Decision Index. WebMCP feed reportsv1.6.29 while page shell saysboardv1.7.46; preserve exact feed revision and source artifact. Intelligence is chance-corrected/weighted. Calibration is a mixed source index, not ECE. Cost and four-axis composite incorporate an explicitly labeled Mercury2.5 proxy tariff because the original7October run was free; this is historical source methodology, not current Mercury Decide pricing. No paid tariff is inferred. Source-computed values are retained; no Examenos recomputation. Per-type support/validity details are in each record. Checked / reported: 2026-10-20 · Samples: 1500 Snapshot: 2026-10-20 · Revision: v1.6.1; source_sha256=e15347783b8bedeeddd9017a831608a957f5bf0386f05902c50161ce0468f911 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
Benchmark Heaven JevBench1.6.1 · Four-axis composite (estimated-cost component)	72.368	Setup & source Benchmark Heaven JevBench1.6.1 · Four-axis composite (estimated-cost component) Benchmark Heaven v1.6.1; measured_in=v1.6.1; named hosted inception/mercury-decide:free.1500decisions (300public+1200sealed); source identity {"model_pin": "provider-reported measurement model slug: inception/mercury-decide:free", "last_measured_on": "2026-10-07", "endpoint_condition": "measured 2026-10-07 on the live v1.6.0/v1.6.1 pool (inputs/input-selfhosted-1500.jsonl, sha256 901983ae...), COMPLETE: 1,500 of 1,500 dispatched, 1,477 answered, 23 invalid. Measured through the release lane's reviewed run_api_full_v161.py UNCHANGED, on Florian's 5 Oct amendment (hosted systems answer the same full S u P as self-hosted ones); provider inception, proxy openrouter, every sealed item exposure-logged before the first request. Florian work order of 6 Oct 2026 ('put it on JevBench', under https://x.com/airesearch12/status/2107569862486659111). NO MAPPING DECISION OF OURS: Mercury Decide publishes the same /v1/systemone contract as Jev and all three primitives come back in the canonical shape (noul a float; choice a label plus probabilities keyed by the item's own option labels; score probabilities keyed by level index '0'..'n-1' - 0-based, unlike ClassOne's), so the vendored typesafe adapter's request body, answer mapping and validations are reused UNCHANGED. or_decisions_r58.OpenRouterDecisionsAdapter rewrites ONLY the URL, because OpenRouter serves the contract at POST /api/alpha/decisions rather than /v1/systemone - its own 400 from chat/completions names that path - and it does so by calling the parent run(), so none of the reviewed mapping is duplicated. The pinned harness/vendor tree was NOT edited; the module is reached by PYTHONPATH. THE 23 LONG ITEMS: on this pool exactly 23 of 1,500 items are 77,857-82,318 tokens and the 24th largest is below 2,048 (run 57's own per-item counts on the identical pool, receipt LONGITEM-TOKENS-R58.json). They are also the 23 items the v1.6.1 cost rule excludes. THIS ROW REFUSES THEM: the context window is 32,768 tokens and all 23 came back as HTTP 422 with the provider's own 'upstream_error: Decision service could not complete the request' - a disclosed refusal, which the harness counts as the system's own rather than as an outage. The invalid set is EXACTLY those 23 and nothing else failed in 1,477 answers. RUN IN TWO PASSES BECAUSE OF A FREE-TIER BURST LIMIT, AND IT DOES NOT MOVE THE SPEED FIGURE: items 1-736 ran back to back at ~1.7 req/s until OpenRouter returned a header-less 429 ('rate_limit_exceeded: Request or spending limit exceeded'), which is run_v16's own stop rule; the row was resumed at 1 req/s and completed. The two passes' medians differ by 0.038 s (0.3624 s vs 0.3246 s over 1,477 answers; 0.3624 vs 0.3104 on the 124 items the Speed axis actually uses), and run_v16 sleeps the spacing BEFORE starting the timer, so the pacing is not in the measurement. Receipt LATENCY-SEGMENTS-mercury-R58.txt. Production API, so NO latency adjustment. Serial, one decision per request, no retry, no batching. add-requests run 58", "adapter_id": null, "measurement_date_status": "run output completion day (UTC)"} Distinct from Leanmcp JevBench and community HF Decision Index. WebMCP feed reportsv1.6.29 while page shell saysboardv1.7.46; preserve exact feed revision and source artifact. Intelligence is chance-corrected/weighted. Calibration is a mixed source index, not ECE. Cost and four-axis composite incorporate an explicitly labeled Mercury2.5 proxy tariff because the original7October run was free; this is historical source methodology, not current Mercury Decide pricing. No paid tariff is inferred. Source-computed values are retained; no Examenos recomputation. Per-type support/validity details are in each record. Distinct from Leanmcp JevBench and community HF Decision Index. WebMCP feed reportsv1.6.29 while page shell saysboardv1.7.46; preserve exact feed revision and source artifact. Intelligence is chance-corrected/weighted. Calibration is a mixed source index, not ECE. Cost and four-axis composite incorporate an explicitly labeled Mercury2.5 proxy tariff because the original7October run was free; this is historical source methodology, not current Mercury Decide pricing. No paid tariff is inferred. Source-computed values are retained; no Examenos recomputation. Per-type support/validity details are in each record. Checked / reported: 2026-10-20 · Samples: 1500 Snapshot: 2026-10-20 · Revision: v1.6.1; source_sha256=e15347783b8bedeeddd9017a831608a957f5bf0386f05902c50161ce0468f911 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
Benchmark Heaven JevBench1.6.1 · I_open	67.129	Setup & source Benchmark Heaven JevBench1.6.1 · I_open Benchmark Heaven v1.6.1; measured_in=v1.6.1; named hosted inception/mercury-decide:free.1500decisions (300public+1200sealed); source identity {"model_pin": "provider-reported measurement model slug: inception/mercury-decide:free", "last_measured_on": "2026-10-07", "endpoint_condition": "measured 2026-10-07 on the live v1.6.0/v1.6.1 pool (inputs/input-selfhosted-1500.jsonl, sha256 901983ae...), COMPLETE: 1,500 of 1,500 dispatched, 1,477 answered, 23 invalid. Measured through the release lane's reviewed run_api_full_v161.py UNCHANGED, on Florian's 5 Oct amendment (hosted systems answer the same full S u P as self-hosted ones); provider inception, proxy openrouter, every sealed item exposure-logged before the first request. Florian work order of 6 Oct 2026 ('put it on JevBench', under https://x.com/airesearch12/status/2107569862486659111). NO MAPPING DECISION OF OURS: Mercury Decide publishes the same /v1/systemone contract as Jev and all three primitives come back in the canonical shape (noul a float; choice a label plus probabilities keyed by the item's own option labels; score probabilities keyed by level index '0'..'n-1' - 0-based, unlike ClassOne's), so the vendored typesafe adapter's request body, answer mapping and validations are reused UNCHANGED. or_decisions_r58.OpenRouterDecisionsAdapter rewrites ONLY the URL, because OpenRouter serves the contract at POST /api/alpha/decisions rather than /v1/systemone - its own 400 from chat/completions names that path - and it does so by calling the parent run(), so none of the reviewed mapping is duplicated. The pinned harness/vendor tree was NOT edited; the module is reached by PYTHONPATH. THE 23 LONG ITEMS: on this pool exactly 23 of 1,500 items are 77,857-82,318 tokens and the 24th largest is below 2,048 (run 57's own per-item counts on the identical pool, receipt LONGITEM-TOKENS-R58.json). They are also the 23 items the v1.6.1 cost rule excludes. THIS ROW REFUSES THEM: the context window is 32,768 tokens and all 23 came back as HTTP 422 with the provider's own 'upstream_error: Decision service could not complete the request' - a disclosed refusal, which the harness counts as the system's own rather than as an outage. The invalid set is EXACTLY those 23 and nothing else failed in 1,477 answers. RUN IN TWO PASSES BECAUSE OF A FREE-TIER BURST LIMIT, AND IT DOES NOT MOVE THE SPEED FIGURE: items 1-736 ran back to back at ~1.7 req/s until OpenRouter returned a header-less 429 ('rate_limit_exceeded: Request or spending limit exceeded'), which is run_v16's own stop rule; the row was resumed at 1 req/s and completed. The two passes' medians differ by 0.038 s (0.3624 s vs 0.3246 s over 1,477 answers; 0.3624 vs 0.3104 on the 124 items the Speed axis actually uses), and run_v16 sleeps the spacing BEFORE starting the timer, so the pacing is not in the measurement. Receipt LATENCY-SEGMENTS-mercury-R58.txt. Production API, so NO latency adjustment. Serial, one decision per request, no retry, no batching. add-requests run 58", "adapter_id": null, "measurement_date_status": "run output completion day (UTC)"} Distinct from Leanmcp JevBench and community HF Decision Index. WebMCP feed reportsv1.6.29 while page shell saysboardv1.7.46; preserve exact feed revision and source artifact. Intelligence is chance-corrected/weighted. Calibration is a mixed source index, not ECE. Cost and four-axis composite incorporate an explicitly labeled Mercury2.5 proxy tariff because the original7October run was free; this is historical source methodology, not current Mercury Decide pricing. No paid tariff is inferred. Source-computed values are retained; no Examenos recomputation. Per-type support/validity details are in each record. Distinct from Leanmcp JevBench and community HF Decision Index. WebMCP feed reportsv1.6.29 while page shell saysboardv1.7.46; preserve exact feed revision and source artifact. Intelligence is chance-corrected/weighted. Calibration is a mixed source index, not ECE. Cost and four-axis composite incorporate an explicitly labeled Mercury2.5 proxy tariff because the original7October run was free; this is historical source methodology, not current Mercury Decide pricing. No paid tariff is inferred. Source-computed values are retained; no Examenos recomputation. Per-type support/validity details are in each record. Checked / reported: 2026-10-20 · Samples: 300 Snapshot: 2026-10-20 · Revision: v1.6.1; source_sha256=e15347783b8bedeeddd9017a831608a957f5bf0386f05902c50161ce0468f911 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
Benchmark Heaven JevBench1.6.1 · I_sealed	64.685	Setup & source Benchmark Heaven JevBench1.6.1 · I_sealed Benchmark Heaven v1.6.1; measured_in=v1.6.1; named hosted inception/mercury-decide:free.1500decisions (300public+1200sealed); source identity {"model_pin": "provider-reported measurement model slug: inception/mercury-decide:free", "last_measured_on": "2026-10-07", "endpoint_condition": "measured 2026-10-07 on the live v1.6.0/v1.6.1 pool (inputs/input-selfhosted-1500.jsonl, sha256 901983ae...), COMPLETE: 1,500 of 1,500 dispatched, 1,477 answered, 23 invalid. Measured through the release lane's reviewed run_api_full_v161.py UNCHANGED, on Florian's 5 Oct amendment (hosted systems answer the same full S u P as self-hosted ones); provider inception, proxy openrouter, every sealed item exposure-logged before the first request. Florian work order of 6 Oct 2026 ('put it on JevBench', under https://x.com/airesearch12/status/2107569862486659111). NO MAPPING DECISION OF OURS: Mercury Decide publishes the same /v1/systemone contract as Jev and all three primitives come back in the canonical shape (noul a float; choice a label plus probabilities keyed by the item's own option labels; score probabilities keyed by level index '0'..'n-1' - 0-based, unlike ClassOne's), so the vendored typesafe adapter's request body, answer mapping and validations are reused UNCHANGED. or_decisions_r58.OpenRouterDecisionsAdapter rewrites ONLY the URL, because OpenRouter serves the contract at POST /api/alpha/decisions rather than /v1/systemone - its own 400 from chat/completions names that path - and it does so by calling the parent run(), so none of the reviewed mapping is duplicated. The pinned harness/vendor tree was NOT edited; the module is reached by PYTHONPATH. THE 23 LONG ITEMS: on this pool exactly 23 of 1,500 items are 77,857-82,318 tokens and the 24th largest is below 2,048 (run 57's own per-item counts on the identical pool, receipt LONGITEM-TOKENS-R58.json). They are also the 23 items the v1.6.1 cost rule excludes. THIS ROW REFUSES THEM: the context window is 32,768 tokens and all 23 came back as HTTP 422 with the provider's own 'upstream_error: Decision service could not complete the request' - a disclosed refusal, which the harness counts as the system's own rather than as an outage. The invalid set is EXACTLY those 23 and nothing else failed in 1,477 answers. RUN IN TWO PASSES BECAUSE OF A FREE-TIER BURST LIMIT, AND IT DOES NOT MOVE THE SPEED FIGURE: items 1-736 ran back to back at ~1.7 req/s until OpenRouter returned a header-less 429 ('rate_limit_exceeded: Request or spending limit exceeded'), which is run_v16's own stop rule; the row was resumed at 1 req/s and completed. The two passes' medians differ by 0.038 s (0.3624 s vs 0.3246 s over 1,477 answers; 0.3624 vs 0.3104 on the 124 items the Speed axis actually uses), and run_v16 sleeps the spacing BEFORE starting the timer, so the pacing is not in the measurement. Receipt LATENCY-SEGMENTS-mercury-R58.txt. Production API, so NO latency adjustment. Serial, one decision per request, no retry, no batching. add-requests run 58", "adapter_id": null, "measurement_date_status": "run output completion day (UTC)"} Distinct from Leanmcp JevBench and community HF Decision Index. WebMCP feed reportsv1.6.29 while page shell saysboardv1.7.46; preserve exact feed revision and source artifact. Intelligence is chance-corrected/weighted. Calibration is a mixed source index, not ECE. Cost and four-axis composite incorporate an explicitly labeled Mercury2.5 proxy tariff because the original7October run was free; this is historical source methodology, not current Mercury Decide pricing. No paid tariff is inferred. Source-computed values are retained; no Examenos recomputation. Per-type support/validity details are in each record. Distinct from Leanmcp JevBench and community HF Decision Index. WebMCP feed reportsv1.6.29 while page shell saysboardv1.7.46; preserve exact feed revision and source artifact. Intelligence is chance-corrected/weighted. Calibration is a mixed source index, not ECE. Cost and four-axis composite incorporate an explicitly labeled Mercury2.5 proxy tariff because the original7October run was free; this is historical source methodology, not current Mercury Decide pricing. No paid tariff is inferred. Source-computed values are retained; no Examenos recomputation. Per-type support/validity details are in each record. Checked / reported: 2026-10-20 · Samples: 1200 Snapshot: 2026-10-20 · Revision: v1.6.1; source_sha256=e15347783b8bedeeddd9017a831608a957f5bf0386f05902c50161ce0468f911 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
Benchmark Heaven JevBench1.6.1 · gap	2.444 pp	Setup & source Benchmark Heaven JevBench1.6.1 · gap Benchmark Heaven v1.6.1; measured_in=v1.6.1; named hosted inception/mercury-decide:free.1500decisions (300public+1200sealed); source identity {"model_pin": "provider-reported measurement model slug: inception/mercury-decide:free", "last_measured_on": "2026-10-07", "endpoint_condition": "measured 2026-10-07 on the live v1.6.0/v1.6.1 pool (inputs/input-selfhosted-1500.jsonl, sha256 901983ae...), COMPLETE: 1,500 of 1,500 dispatched, 1,477 answered, 23 invalid. Measured through the release lane's reviewed run_api_full_v161.py UNCHANGED, on Florian's 5 Oct amendment (hosted systems answer the same full S u P as self-hosted ones); provider inception, proxy openrouter, every sealed item exposure-logged before the first request. Florian work order of 6 Oct 2026 ('put it on JevBench', under https://x.com/airesearch12/status/2107569862486659111). NO MAPPING DECISION OF OURS: Mercury Decide publishes the same /v1/systemone contract as Jev and all three primitives come back in the canonical shape (noul a float; choice a label plus probabilities keyed by the item's own option labels; score probabilities keyed by level index '0'..'n-1' - 0-based, unlike ClassOne's), so the vendored typesafe adapter's request body, answer mapping and validations are reused UNCHANGED. or_decisions_r58.OpenRouterDecisionsAdapter rewrites ONLY the URL, because OpenRouter serves the contract at POST /api/alpha/decisions rather than /v1/systemone - its own 400 from chat/completions names that path - and it does so by calling the parent run(), so none of the reviewed mapping is duplicated. The pinned harness/vendor tree was NOT edited; the module is reached by PYTHONPATH. THE 23 LONG ITEMS: on this pool exactly 23 of 1,500 items are 77,857-82,318 tokens and the 24th largest is below 2,048 (run 57's own per-item counts on the identical pool, receipt LONGITEM-TOKENS-R58.json). They are also the 23 items the v1.6.1 cost rule excludes. THIS ROW REFUSES THEM: the context window is 32,768 tokens and all 23 came back as HTTP 422 with the provider's own 'upstream_error: Decision service could not complete the request' - a disclosed refusal, which the harness counts as the system's own rather than as an outage. The invalid set is EXACTLY those 23 and nothing else failed in 1,477 answers. RUN IN TWO PASSES BECAUSE OF A FREE-TIER BURST LIMIT, AND IT DOES NOT MOVE THE SPEED FIGURE: items 1-736 ran back to back at ~1.7 req/s until OpenRouter returned a header-less 429 ('rate_limit_exceeded: Request or spending limit exceeded'), which is run_v16's own stop rule; the row was resumed at 1 req/s and completed. The two passes' medians differ by 0.038 s (0.3624 s vs 0.3246 s over 1,477 answers; 0.3624 vs 0.3104 on the 124 items the Speed axis actually uses), and run_v16 sleeps the spacing BEFORE starting the timer, so the pacing is not in the measurement. Receipt LATENCY-SEGMENTS-mercury-R58.txt. Production API, so NO latency adjustment. Serial, one decision per request, no retry, no batching. add-requests run 58", "adapter_id": null, "measurement_date_status": "run output completion day (UTC)"} Distinct from Leanmcp JevBench and community HF Decision Index. WebMCP feed reportsv1.6.29 while page shell saysboardv1.7.46; preserve exact feed revision and source artifact. Intelligence is chance-corrected/weighted. Calibration is a mixed source index, not ECE. Cost and four-axis composite incorporate an explicitly labeled Mercury2.5 proxy tariff because the original7October run was free; this is historical source methodology, not current Mercury Decide pricing. No paid tariff is inferred. Source-computed values are retained; no Examenos recomputation. Per-type support/validity details are in each record. Distinct from Leanmcp JevBench and community HF Decision Index. WebMCP feed reportsv1.6.29 while page shell saysboardv1.7.46; preserve exact feed revision and source artifact. Intelligence is chance-corrected/weighted. Calibration is a mixed source index, not ECE. Cost and four-axis composite incorporate an explicitly labeled Mercury2.5 proxy tariff because the original7October run was free; this is historical source methodology, not current Mercury Decide pricing. No paid tariff is inferred. Source-computed values are retained; no Examenos recomputation. Per-type support/validity details are in each record. Checked / reported: 2026-10-20 · Samples: 1500 Snapshot: 2026-10-20 · Revision: v1.6.1; source_sha256=e15347783b8bedeeddd9017a831608a957f5bf0386f05902c50161ce0468f911 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
Benchmark Heaven JevBench1.6.1 · Calibration · choice · score	77.909	Setup & source Benchmark Heaven JevBench1.6.1 · Calibration · choice · score Benchmark Heaven v1.6.1; measured_in=v1.6.1; named hosted inception/mercury-decide:free.1500decisions (300public+1200sealed); source identity {"model_pin": "provider-reported measurement model slug: inception/mercury-decide:free", "last_measured_on": "2026-10-07", "endpoint_condition": "measured 2026-10-07 on the live v1.6.0/v1.6.1 pool (inputs/input-selfhosted-1500.jsonl, sha256 901983ae...), COMPLETE: 1,500 of 1,500 dispatched, 1,477 answered, 23 invalid. Measured through the release lane's reviewed run_api_full_v161.py UNCHANGED, on Florian's 5 Oct amendment (hosted systems answer the same full S u P as self-hosted ones); provider inception, proxy openrouter, every sealed item exposure-logged before the first request. Florian work order of 6 Oct 2026 ('put it on JevBench', under https://x.com/airesearch12/status/2107569862486659111). NO MAPPING DECISION OF OURS: Mercury Decide publishes the same /v1/systemone contract as Jev and all three primitives come back in the canonical shape (noul a float; choice a label plus probabilities keyed by the item's own option labels; score probabilities keyed by level index '0'..'n-1' - 0-based, unlike ClassOne's), so the vendored typesafe adapter's request body, answer mapping and validations are reused UNCHANGED. or_decisions_r58.OpenRouterDecisionsAdapter rewrites ONLY the URL, because OpenRouter serves the contract at POST /api/alpha/decisions rather than /v1/systemone - its own 400 from chat/completions names that path - and it does so by calling the parent run(), so none of the reviewed mapping is duplicated. The pinned harness/vendor tree was NOT edited; the module is reached by PYTHONPATH. THE 23 LONG ITEMS: on this pool exactly 23 of 1,500 items are 77,857-82,318 tokens and the 24th largest is below 2,048 (run 57's own per-item counts on the identical pool, receipt LONGITEM-TOKENS-R58.json). They are also the 23 items the v1.6.1 cost rule excludes. THIS ROW REFUSES THEM: the context window is 32,768 tokens and all 23 came back as HTTP 422 with the provider's own 'upstream_error: Decision service could not complete the request' - a disclosed refusal, which the harness counts as the system's own rather than as an outage. The invalid set is EXACTLY those 23 and nothing else failed in 1,477 answers. RUN IN TWO PASSES BECAUSE OF A FREE-TIER BURST LIMIT, AND IT DOES NOT MOVE THE SPEED FIGURE: items 1-736 ran back to back at ~1.7 req/s until OpenRouter returned a header-less 429 ('rate_limit_exceeded: Request or spending limit exceeded'), which is run_v16's own stop rule; the row was resumed at 1 req/s and completed. The two passes' medians differ by 0.038 s (0.3624 s vs 0.3246 s over 1,477 answers; 0.3624 vs 0.3104 on the 124 items the Speed axis actually uses), and run_v16 sleeps the spacing BEFORE starting the timer, so the pacing is not in the measurement. Receipt LATENCY-SEGMENTS-mercury-R58.txt. Production API, so NO latency adjustment. Serial, one decision per request, no retry, no batching. add-requests run 58", "adapter_id": null, "measurement_date_status": "run output completion day (UTC)"} Distinct from Leanmcp JevBench and community HF Decision Index. WebMCP feed reportsv1.6.29 while page shell saysboardv1.7.46; preserve exact feed revision and source artifact. Intelligence is chance-corrected/weighted. Calibration is a mixed source index, not ECE. Cost and four-axis composite incorporate an explicitly labeled Mercury2.5 proxy tariff because the original7October run was free; this is historical source methodology, not current Mercury Decide pricing. No paid tariff is inferred. Source-computed values are retained; no Examenos recomputation. Per-type support/validity details are in each record. Distinct from Leanmcp JevBench and community HF Decision Index. WebMCP feed reportsv1.6.29 while page shell saysboardv1.7.46; preserve exact feed revision and source artifact. Intelligence is chance-corrected/weighted. Calibration is a mixed source index, not ECE. Cost and four-axis composite incorporate an explicitly labeled Mercury2.5 proxy tariff because the original7October run was free; this is historical source methodology, not current Mercury Decide pricing. No paid tariff is inferred. Source-computed values are retained; no Examenos recomputation. Per-type support/validity details are in each record. Checked / reported: 2026-10-20 · Samples: 737 Snapshot: 2026-10-20 · Revision: v1.6.1; source_sha256=e15347783b8bedeeddd9017a831608a957f5bf0386f05902c50161ce0468f911 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
Benchmark Heaven JevBench1.6.1 · Calibration · choice · ece_hard	11.045 pp	Setup & source Benchmark Heaven JevBench1.6.1 · Calibration · choice · ece_hard Benchmark Heaven v1.6.1; measured_in=v1.6.1; named hosted inception/mercury-decide:free.1500decisions (300public+1200sealed); source identity {"model_pin": "provider-reported measurement model slug: inception/mercury-decide:free", "last_measured_on": "2026-10-07", "endpoint_condition": "measured 2026-10-07 on the live v1.6.0/v1.6.1 pool (inputs/input-selfhosted-1500.jsonl, sha256 901983ae...), COMPLETE: 1,500 of 1,500 dispatched, 1,477 answered, 23 invalid. Measured through the release lane's reviewed run_api_full_v161.py UNCHANGED, on Florian's 5 Oct amendment (hosted systems answer the same full S u P as self-hosted ones); provider inception, proxy openrouter, every sealed item exposure-logged before the first request. Florian work order of 6 Oct 2026 ('put it on JevBench', under https://x.com/airesearch12/status/2107569862486659111). NO MAPPING DECISION OF OURS: Mercury Decide publishes the same /v1/systemone contract as Jev and all three primitives come back in the canonical shape (noul a float; choice a label plus probabilities keyed by the item's own option labels; score probabilities keyed by level index '0'..'n-1' - 0-based, unlike ClassOne's), so the vendored typesafe adapter's request body, answer mapping and validations are reused UNCHANGED. or_decisions_r58.OpenRouterDecisionsAdapter rewrites ONLY the URL, because OpenRouter serves the contract at POST /api/alpha/decisions rather than /v1/systemone - its own 400 from chat/completions names that path - and it does so by calling the parent run(), so none of the reviewed mapping is duplicated. The pinned harness/vendor tree was NOT edited; the module is reached by PYTHONPATH. THE 23 LONG ITEMS: on this pool exactly 23 of 1,500 items are 77,857-82,318 tokens and the 24th largest is below 2,048 (run 57's own per-item counts on the identical pool, receipt LONGITEM-TOKENS-R58.json). They are also the 23 items the v1.6.1 cost rule excludes. THIS ROW REFUSES THEM: the context window is 32,768 tokens and all 23 came back as HTTP 422 with the provider's own 'upstream_error: Decision service could not complete the request' - a disclosed refusal, which the harness counts as the system's own rather than as an outage. The invalid set is EXACTLY those 23 and nothing else failed in 1,477 answers. RUN IN TWO PASSES BECAUSE OF A FREE-TIER BURST LIMIT, AND IT DOES NOT MOVE THE SPEED FIGURE: items 1-736 ran back to back at ~1.7 req/s until OpenRouter returned a header-less 429 ('rate_limit_exceeded: Request or spending limit exceeded'), which is run_v16's own stop rule; the row was resumed at 1 req/s and completed. The two passes' medians differ by 0.038 s (0.3624 s vs 0.3246 s over 1,477 answers; 0.3624 vs 0.3104 on the 124 items the Speed axis actually uses), and run_v16 sleeps the spacing BEFORE starting the timer, so the pacing is not in the measurement. Receipt LATENCY-SEGMENTS-mercury-R58.txt. Production API, so NO latency adjustment. Serial, one decision per request, no retry, no batching. add-requests run 58", "adapter_id": null, "measurement_date_status": "run output completion day (UTC)"} Distinct from Leanmcp JevBench and community HF Decision Index. WebMCP feed reportsv1.6.29 while page shell saysboardv1.7.46; preserve exact feed revision and source artifact. Intelligence is chance-corrected/weighted. Calibration is a mixed source index, not ECE. Cost and four-axis composite incorporate an explicitly labeled Mercury2.5 proxy tariff because the original7October run was free; this is historical source methodology, not current Mercury Decide pricing. No paid tariff is inferred. Source-computed values are retained; no Examenos recomputation. Per-type support/validity details are in each record. Distinct from Leanmcp JevBench and community HF Decision Index. WebMCP feed reportsv1.6.29 while page shell saysboardv1.7.46; preserve exact feed revision and source artifact. Intelligence is chance-corrected/weighted. Calibration is a mixed source index, not ECE. Cost and four-axis composite incorporate an explicitly labeled Mercury2.5 proxy tariff because the original7October run was free; this is historical source methodology, not current Mercury Decide pricing. No paid tariff is inferred. Source-computed values are retained; no Examenos recomputation. Per-type support/validity details are in each record. Checked / reported: 2026-10-20 · Samples: 737 Snapshot: 2026-10-20 · Revision: v1.6.1; source_sha256=e15347783b8bedeeddd9017a831608a957f5bf0386f05902c50161ce0468f911 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
Benchmark Heaven JevBench1.6.1 · Calibration · noul · score	82.372	Setup & source Benchmark Heaven JevBench1.6.1 · Calibration · noul · score Benchmark Heaven v1.6.1; measured_in=v1.6.1; named hosted inception/mercury-decide:free.1500decisions (300public+1200sealed); source identity {"model_pin": "provider-reported measurement model slug: inception/mercury-decide:free", "last_measured_on": "2026-10-07", "endpoint_condition": "measured 2026-10-07 on the live v1.6.0/v1.6.1 pool (inputs/input-selfhosted-1500.jsonl, sha256 901983ae...), COMPLETE: 1,500 of 1,500 dispatched, 1,477 answered, 23 invalid. Measured through the release lane's reviewed run_api_full_v161.py UNCHANGED, on Florian's 5 Oct amendment (hosted systems answer the same full S u P as self-hosted ones); provider inception, proxy openrouter, every sealed item exposure-logged before the first request. Florian work order of 6 Oct 2026 ('put it on JevBench', under https://x.com/airesearch12/status/2107569862486659111). NO MAPPING DECISION OF OURS: Mercury Decide publishes the same /v1/systemone contract as Jev and all three primitives come back in the canonical shape (noul a float; choice a label plus probabilities keyed by the item's own option labels; score probabilities keyed by level index '0'..'n-1' - 0-based, unlike ClassOne's), so the vendored typesafe adapter's request body, answer mapping and validations are reused UNCHANGED. or_decisions_r58.OpenRouterDecisionsAdapter rewrites ONLY the URL, because OpenRouter serves the contract at POST /api/alpha/decisions rather than /v1/systemone - its own 400 from chat/completions names that path - and it does so by calling the parent run(), so none of the reviewed mapping is duplicated. The pinned harness/vendor tree was NOT edited; the module is reached by PYTHONPATH. THE 23 LONG ITEMS: on this pool exactly 23 of 1,500 items are 77,857-82,318 tokens and the 24th largest is below 2,048 (run 57's own per-item counts on the identical pool, receipt LONGITEM-TOKENS-R58.json). They are also the 23 items the v1.6.1 cost rule excludes. THIS ROW REFUSES THEM: the context window is 32,768 tokens and all 23 came back as HTTP 422 with the provider's own 'upstream_error: Decision service could not complete the request' - a disclosed refusal, which the harness counts as the system's own rather than as an outage. The invalid set is EXACTLY those 23 and nothing else failed in 1,477 answers. RUN IN TWO PASSES BECAUSE OF A FREE-TIER BURST LIMIT, AND IT DOES NOT MOVE THE SPEED FIGURE: items 1-736 ran back to back at ~1.7 req/s until OpenRouter returned a header-less 429 ('rate_limit_exceeded: Request or spending limit exceeded'), which is run_v16's own stop rule; the row was resumed at 1 req/s and completed. The two passes' medians differ by 0.038 s (0.3624 s vs 0.3246 s over 1,477 answers; 0.3624 vs 0.3104 on the 124 items the Speed axis actually uses), and run_v16 sleeps the spacing BEFORE starting the timer, so the pacing is not in the measurement. Receipt LATENCY-SEGMENTS-mercury-R58.txt. Production API, so NO latency adjustment. Serial, one decision per request, no retry, no batching. add-requests run 58", "adapter_id": null, "measurement_date_status": "run output completion day (UTC)"} Distinct from Leanmcp JevBench and community HF Decision Index. WebMCP feed reportsv1.6.29 while page shell saysboardv1.7.46; preserve exact feed revision and source artifact. Intelligence is chance-corrected/weighted. Calibration is a mixed source index, not ECE. Cost and four-axis composite incorporate an explicitly labeled Mercury2.5 proxy tariff because the original7October run was free; this is historical source methodology, not current Mercury Decide pricing. No paid tariff is inferred. Source-computed values are retained; no Examenos recomputation. Per-type support/validity details are in each record. Distinct from Leanmcp JevBench and community HF Decision Index. WebMCP feed reportsv1.6.29 while page shell saysboardv1.7.46; preserve exact feed revision and source artifact. Intelligence is chance-corrected/weighted. Calibration is a mixed source index, not ECE. Cost and four-axis composite incorporate an explicitly labeled Mercury2.5 proxy tariff because the original7October run was free; this is historical source methodology, not current Mercury Decide pricing. No paid tariff is inferred. Source-computed values are retained; no Examenos recomputation. Per-type support/validity details are in each record. Checked / reported: 2026-10-20 · Samples: 371 Snapshot: 2026-10-20 · Revision: v1.6.1; source_sha256=e15347783b8bedeeddd9017a831608a957f5bf0386f05902c50161ce0468f911 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
Benchmark Heaven JevBench1.6.1 · Calibration · noul · brier	0.101512 Brier	Setup & source Benchmark Heaven JevBench1.6.1 · Calibration · noul · brier Benchmark Heaven v1.6.1; measured_in=v1.6.1; named hosted inception/mercury-decide:free.1500decisions (300public+1200sealed); source identity {"model_pin": "provider-reported measurement model slug: inception/mercury-decide:free", "last_measured_on": "2026-10-07", "endpoint_condition": "measured 2026-10-07 on the live v1.6.0/v1.6.1 pool (inputs/input-selfhosted-1500.jsonl, sha256 901983ae...), COMPLETE: 1,500 of 1,500 dispatched, 1,477 answered, 23 invalid. Measured through the release lane's reviewed run_api_full_v161.py UNCHANGED, on Florian's 5 Oct amendment (hosted systems answer the same full S u P as self-hosted ones); provider inception, proxy openrouter, every sealed item exposure-logged before the first request. Florian work order of 6 Oct 2026 ('put it on JevBench', under https://x.com/airesearch12/status/2107569862486659111). NO MAPPING DECISION OF OURS: Mercury Decide publishes the same /v1/systemone contract as Jev and all three primitives come back in the canonical shape (noul a float; choice a label plus probabilities keyed by the item's own option labels; score probabilities keyed by level index '0'..'n-1' - 0-based, unlike ClassOne's), so the vendored typesafe adapter's request body, answer mapping and validations are reused UNCHANGED. or_decisions_r58.OpenRouterDecisionsAdapter rewrites ONLY the URL, because OpenRouter serves the contract at POST /api/alpha/decisions rather than /v1/systemone - its own 400 from chat/completions names that path - and it does so by calling the parent run(), so none of the reviewed mapping is duplicated. The pinned harness/vendor tree was NOT edited; the module is reached by PYTHONPATH. THE 23 LONG ITEMS: on this pool exactly 23 of 1,500 items are 77,857-82,318 tokens and the 24th largest is below 2,048 (run 57's own per-item counts on the identical pool, receipt LONGITEM-TOKENS-R58.json). They are also the 23 items the v1.6.1 cost rule excludes. THIS ROW REFUSES THEM: the context window is 32,768 tokens and all 23 came back as HTTP 422 with the provider's own 'upstream_error: Decision service could not complete the request' - a disclosed refusal, which the harness counts as the system's own rather than as an outage. The invalid set is EXACTLY those 23 and nothing else failed in 1,477 answers. RUN IN TWO PASSES BECAUSE OF A FREE-TIER BURST LIMIT, AND IT DOES NOT MOVE THE SPEED FIGURE: items 1-736 ran back to back at ~1.7 req/s until OpenRouter returned a header-less 429 ('rate_limit_exceeded: Request or spending limit exceeded'), which is run_v16's own stop rule; the row was resumed at 1 req/s and completed. The two passes' medians differ by 0.038 s (0.3624 s vs 0.3246 s over 1,477 answers; 0.3624 vs 0.3104 on the 124 items the Speed axis actually uses), and run_v16 sleeps the spacing BEFORE starting the timer, so the pacing is not in the measurement. Receipt LATENCY-SEGMENTS-mercury-R58.txt. Production API, so NO latency adjustment. Serial, one decision per request, no retry, no batching. add-requests run 58", "adapter_id": null, "measurement_date_status": "run output completion day (UTC)"} Distinct from Leanmcp JevBench and community HF Decision Index. WebMCP feed reportsv1.6.29 while page shell saysboardv1.7.46; preserve exact feed revision and source artifact. Intelligence is chance-corrected/weighted. Calibration is a mixed source index, not ECE. Cost and four-axis composite incorporate an explicitly labeled Mercury2.5 proxy tariff because the original7October run was free; this is historical source methodology, not current Mercury Decide pricing. No paid tariff is inferred. Source-computed values are retained; no Examenos recomputation. Per-type support/validity details are in each record. Distinct from Leanmcp JevBench and community HF Decision Index. WebMCP feed reportsv1.6.29 while page shell saysboardv1.7.46; preserve exact feed revision and source artifact. Intelligence is chance-corrected/weighted. Calibration is a mixed source index, not ECE. Cost and four-axis composite incorporate an explicitly labeled Mercury2.5 proxy tariff because the original7October run was free; this is historical source methodology, not current Mercury Decide pricing. No paid tariff is inferred. Source-computed values are retained; no Examenos recomputation. Per-type support/validity details are in each record. Checked / reported: 2026-10-20 · Samples: 371 Snapshot: 2026-10-20 · Revision: v1.6.1; source_sha256=e15347783b8bedeeddd9017a831608a957f5bf0386f05902c50161ce0468f911 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
Benchmark Heaven JevBench1.6.1 · Calibration · noul · ece	8.814 pp	Setup & source Benchmark Heaven JevBench1.6.1 · Calibration · noul · ece Benchmark Heaven v1.6.1; measured_in=v1.6.1; named hosted inception/mercury-decide:free.1500decisions (300public+1200sealed); source identity {"model_pin": "provider-reported measurement model slug: inception/mercury-decide:free", "last_measured_on": "2026-10-07", "endpoint_condition": "measured 2026-10-07 on the live v1.6.0/v1.6.1 pool (inputs/input-selfhosted-1500.jsonl, sha256 901983ae...), COMPLETE: 1,500 of 1,500 dispatched, 1,477 answered, 23 invalid. Measured through the release lane's reviewed run_api_full_v161.py UNCHANGED, on Florian's 5 Oct amendment (hosted systems answer the same full S u P as self-hosted ones); provider inception, proxy openrouter, every sealed item exposure-logged before the first request. Florian work order of 6 Oct 2026 ('put it on JevBench', under https://x.com/airesearch12/status/2107569862486659111). NO MAPPING DECISION OF OURS: Mercury Decide publishes the same /v1/systemone contract as Jev and all three primitives come back in the canonical shape (noul a float; choice a label plus probabilities keyed by the item's own option labels; score probabilities keyed by level index '0'..'n-1' - 0-based, unlike ClassOne's), so the vendored typesafe adapter's request body, answer mapping and validations are reused UNCHANGED. or_decisions_r58.OpenRouterDecisionsAdapter rewrites ONLY the URL, because OpenRouter serves the contract at POST /api/alpha/decisions rather than /v1/systemone - its own 400 from chat/completions names that path - and it does so by calling the parent run(), so none of the reviewed mapping is duplicated. The pinned harness/vendor tree was NOT edited; the module is reached by PYTHONPATH. THE 23 LONG ITEMS: on this pool exactly 23 of 1,500 items are 77,857-82,318 tokens and the 24th largest is below 2,048 (run 57's own per-item counts on the identical pool, receipt LONGITEM-TOKENS-R58.json). They are also the 23 items the v1.6.1 cost rule excludes. THIS ROW REFUSES THEM: the context window is 32,768 tokens and all 23 came back as HTTP 422 with the provider's own 'upstream_error: Decision service could not complete the request' - a disclosed refusal, which the harness counts as the system's own rather than as an outage. The invalid set is EXACTLY those 23 and nothing else failed in 1,477 answers. RUN IN TWO PASSES BECAUSE OF A FREE-TIER BURST LIMIT, AND IT DOES NOT MOVE THE SPEED FIGURE: items 1-736 ran back to back at ~1.7 req/s until OpenRouter returned a header-less 429 ('rate_limit_exceeded: Request or spending limit exceeded'), which is run_v16's own stop rule; the row was resumed at 1 req/s and completed. The two passes' medians differ by 0.038 s (0.3624 s vs 0.3246 s over 1,477 answers; 0.3624 vs 0.3104 on the 124 items the Speed axis actually uses), and run_v16 sleeps the spacing BEFORE starting the timer, so the pacing is not in the measurement. Receipt LATENCY-SEGMENTS-mercury-R58.txt. Production API, so NO latency adjustment. Serial, one decision per request, no retry, no batching. add-requests run 58", "adapter_id": null, "measurement_date_status": "run output completion day (UTC)"} Distinct from Leanmcp JevBench and community HF Decision Index. WebMCP feed reportsv1.6.29 while page shell saysboardv1.7.46; preserve exact feed revision and source artifact. Intelligence is chance-corrected/weighted. Calibration is a mixed source index, not ECE. Cost and four-axis composite incorporate an explicitly labeled Mercury2.5 proxy tariff because the original7October run was free; this is historical source methodology, not current Mercury Decide pricing. No paid tariff is inferred. Source-computed values are retained; no Examenos recomputation. Per-type support/validity details are in each record. Distinct from Leanmcp JevBench and community HF Decision Index. WebMCP feed reportsv1.6.29 while page shell saysboardv1.7.46; preserve exact feed revision and source artifact. Intelligence is chance-corrected/weighted. Calibration is a mixed source index, not ECE. Cost and four-axis composite incorporate an explicitly labeled Mercury2.5 proxy tariff because the original7October run was free; this is historical source methodology, not current Mercury Decide pricing. No paid tariff is inferred. Source-computed values are retained; no Examenos recomputation. Per-type support/validity details are in each record. Checked / reported: 2026-10-20 · Samples: 371 Snapshot: 2026-10-20 · Revision: v1.6.1; source_sha256=e15347783b8bedeeddd9017a831608a957f5bf0386f05902c50161ce0468f911 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
Benchmark Heaven JevBench1.6.1 · Calibration · noul · abstention_rate	10.243%	Setup & source Benchmark Heaven JevBench1.6.1 · Calibration · noul · abstention_rate Benchmark Heaven v1.6.1; measured_in=v1.6.1; named hosted inception/mercury-decide:free.1500decisions (300public+1200sealed); source identity {"model_pin": "provider-reported measurement model slug: inception/mercury-decide:free", "last_measured_on": "2026-10-07", "endpoint_condition": "measured 2026-10-07 on the live v1.6.0/v1.6.1 pool (inputs/input-selfhosted-1500.jsonl, sha256 901983ae...), COMPLETE: 1,500 of 1,500 dispatched, 1,477 answered, 23 invalid. Measured through the release lane's reviewed run_api_full_v161.py UNCHANGED, on Florian's 5 Oct amendment (hosted systems answer the same full S u P as self-hosted ones); provider inception, proxy openrouter, every sealed item exposure-logged before the first request. Florian work order of 6 Oct 2026 ('put it on JevBench', under https://x.com/airesearch12/status/2107569862486659111). NO MAPPING DECISION OF OURS: Mercury Decide publishes the same /v1/systemone contract as Jev and all three primitives come back in the canonical shape (noul a float; choice a label plus probabilities keyed by the item's own option labels; score probabilities keyed by level index '0'..'n-1' - 0-based, unlike ClassOne's), so the vendored typesafe adapter's request body, answer mapping and validations are reused UNCHANGED. or_decisions_r58.OpenRouterDecisionsAdapter rewrites ONLY the URL, because OpenRouter serves the contract at POST /api/alpha/decisions rather than /v1/systemone - its own 400 from chat/completions names that path - and it does so by calling the parent run(), so none of the reviewed mapping is duplicated. The pinned harness/vendor tree was NOT edited; the module is reached by PYTHONPATH. THE 23 LONG ITEMS: on this pool exactly 23 of 1,500 items are 77,857-82,318 tokens and the 24th largest is below 2,048 (run 57's own per-item counts on the identical pool, receipt LONGITEM-TOKENS-R58.json). They are also the 23 items the v1.6.1 cost rule excludes. THIS ROW REFUSES THEM: the context window is 32,768 tokens and all 23 came back as HTTP 422 with the provider's own 'upstream_error: Decision service could not complete the request' - a disclosed refusal, which the harness counts as the system's own rather than as an outage. The invalid set is EXACTLY those 23 and nothing else failed in 1,477 answers. RUN IN TWO PASSES BECAUSE OF A FREE-TIER BURST LIMIT, AND IT DOES NOT MOVE THE SPEED FIGURE: items 1-736 ran back to back at ~1.7 req/s until OpenRouter returned a header-less 429 ('rate_limit_exceeded: Request or spending limit exceeded'), which is run_v16's own stop rule; the row was resumed at 1 req/s and completed. The two passes' medians differ by 0.038 s (0.3624 s vs 0.3246 s over 1,477 answers; 0.3624 vs 0.3104 on the 124 items the Speed axis actually uses), and run_v16 sleeps the spacing BEFORE starting the timer, so the pacing is not in the measurement. Receipt LATENCY-SEGMENTS-mercury-R58.txt. Production API, so NO latency adjustment. Serial, one decision per request, no retry, no batching. add-requests run 58", "adapter_id": null, "measurement_date_status": "run output completion day (UTC)"} Distinct from Leanmcp JevBench and community HF Decision Index. WebMCP feed reportsv1.6.29 while page shell saysboardv1.7.46; preserve exact feed revision and source artifact. Intelligence is chance-corrected/weighted. Calibration is a mixed source index, not ECE. Cost and four-axis composite incorporate an explicitly labeled Mercury2.5 proxy tariff because the original7October run was free; this is historical source methodology, not current Mercury Decide pricing. No paid tariff is inferred. Source-computed values are retained; no Examenos recomputation. Per-type support/validity details are in each record. Distinct from Leanmcp JevBench and community HF Decision Index. WebMCP feed reportsv1.6.29 while page shell saysboardv1.7.46; preserve exact feed revision and source artifact. Intelligence is chance-corrected/weighted. Calibration is a mixed source index, not ECE. Cost and four-axis composite incorporate an explicitly labeled Mercury2.5 proxy tariff because the original7October run was free; this is historical source methodology, not current Mercury Decide pricing. No paid tariff is inferred. Source-computed values are retained; no Examenos recomputation. Per-type support/validity details are in each record. Checked / reported: 2026-10-20 · Samples: 371 Snapshot: 2026-10-20 · Revision: v1.6.1; source_sha256=e15347783b8bedeeddd9017a831608a957f5bf0386f05902c50161ce0468f911 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
Benchmark Heaven JevBench1.6.1 · Calibration · score · score	83.749	Setup & source Benchmark Heaven JevBench1.6.1 · Calibration · score · score Benchmark Heaven v1.6.1; measured_in=v1.6.1; named hosted inception/mercury-decide:free.1500decisions (300public+1200sealed); source identity {"model_pin": "provider-reported measurement model slug: inception/mercury-decide:free", "last_measured_on": "2026-10-07", "endpoint_condition": "measured 2026-10-07 on the live v1.6.0/v1.6.1 pool (inputs/input-selfhosted-1500.jsonl, sha256 901983ae...), COMPLETE: 1,500 of 1,500 dispatched, 1,477 answered, 23 invalid. Measured through the release lane's reviewed run_api_full_v161.py UNCHANGED, on Florian's 5 Oct amendment (hosted systems answer the same full S u P as self-hosted ones); provider inception, proxy openrouter, every sealed item exposure-logged before the first request. Florian work order of 6 Oct 2026 ('put it on JevBench', under https://x.com/airesearch12/status/2107569862486659111). NO MAPPING DECISION OF OURS: Mercury Decide publishes the same /v1/systemone contract as Jev and all three primitives come back in the canonical shape (noul a float; choice a label plus probabilities keyed by the item's own option labels; score probabilities keyed by level index '0'..'n-1' - 0-based, unlike ClassOne's), so the vendored typesafe adapter's request body, answer mapping and validations are reused UNCHANGED. or_decisions_r58.OpenRouterDecisionsAdapter rewrites ONLY the URL, because OpenRouter serves the contract at POST /api/alpha/decisions rather than /v1/systemone - its own 400 from chat/completions names that path - and it does so by calling the parent run(), so none of the reviewed mapping is duplicated. The pinned harness/vendor tree was NOT edited; the module is reached by PYTHONPATH. THE 23 LONG ITEMS: on this pool exactly 23 of 1,500 items are 77,857-82,318 tokens and the 24th largest is below 2,048 (run 57's own per-item counts on the identical pool, receipt LONGITEM-TOKENS-R58.json). They are also the 23 items the v1.6.1 cost rule excludes. THIS ROW REFUSES THEM: the context window is 32,768 tokens and all 23 came back as HTTP 422 with the provider's own 'upstream_error: Decision service could not complete the request' - a disclosed refusal, which the harness counts as the system's own rather than as an outage. The invalid set is EXACTLY those 23 and nothing else failed in 1,477 answers. RUN IN TWO PASSES BECAUSE OF A FREE-TIER BURST LIMIT, AND IT DOES NOT MOVE THE SPEED FIGURE: items 1-736 ran back to back at ~1.7 req/s until OpenRouter returned a header-less 429 ('rate_limit_exceeded: Request or spending limit exceeded'), which is run_v16's own stop rule; the row was resumed at 1 req/s and completed. The two passes' medians differ by 0.038 s (0.3624 s vs 0.3246 s over 1,477 answers; 0.3624 vs 0.3104 on the 124 items the Speed axis actually uses), and run_v16 sleeps the spacing BEFORE starting the timer, so the pacing is not in the measurement. Receipt LATENCY-SEGMENTS-mercury-R58.txt. Production API, so NO latency adjustment. Serial, one decision per request, no retry, no batching. add-requests run 58", "adapter_id": null, "measurement_date_status": "run output completion day (UTC)"} Distinct from Leanmcp JevBench and community HF Decision Index. WebMCP feed reportsv1.6.29 while page shell saysboardv1.7.46; preserve exact feed revision and source artifact. Intelligence is chance-corrected/weighted. Calibration is a mixed source index, not ECE. Cost and four-axis composite incorporate an explicitly labeled Mercury2.5 proxy tariff because the original7October run was free; this is historical source methodology, not current Mercury Decide pricing. No paid tariff is inferred. Source-computed values are retained; no Examenos recomputation. Per-type support/validity details are in each record. Distinct from Leanmcp JevBench and community HF Decision Index. WebMCP feed reportsv1.6.29 while page shell saysboardv1.7.46; preserve exact feed revision and source artifact. Intelligence is chance-corrected/weighted. Calibration is a mixed source index, not ECE. Cost and four-axis composite incorporate an explicitly labeled Mercury2.5 proxy tariff because the original7October run was free; this is historical source methodology, not current Mercury Decide pricing. No paid tariff is inferred. Source-computed values are retained; no Examenos recomputation. Per-type support/validity details are in each record. Checked / reported: 2026-10-20 · Samples: 369 Snapshot: 2026-10-20 · Revision: v1.6.1; source_sha256=e15347783b8bedeeddd9017a831608a957f5bf0386f05902c50161ce0468f911 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
Benchmark Heaven JevBench1.6.1 · Calibration · score · nrps	0.062397 RPS	Setup & source Benchmark Heaven JevBench1.6.1 · Calibration · score · nrps Benchmark Heaven v1.6.1; measured_in=v1.6.1; named hosted inception/mercury-decide:free.1500decisions (300public+1200sealed); source identity {"model_pin": "provider-reported measurement model slug: inception/mercury-decide:free", "last_measured_on": "2026-10-07", "endpoint_condition": "measured 2026-10-07 on the live v1.6.0/v1.6.1 pool (inputs/input-selfhosted-1500.jsonl, sha256 901983ae...), COMPLETE: 1,500 of 1,500 dispatched, 1,477 answered, 23 invalid. Measured through the release lane's reviewed run_api_full_v161.py UNCHANGED, on Florian's 5 Oct amendment (hosted systems answer the same full S u P as self-hosted ones); provider inception, proxy openrouter, every sealed item exposure-logged before the first request. Florian work order of 6 Oct 2026 ('put it on JevBench', under https://x.com/airesearch12/status/2107569862486659111). NO MAPPING DECISION OF OURS: Mercury Decide publishes the same /v1/systemone contract as Jev and all three primitives come back in the canonical shape (noul a float; choice a label plus probabilities keyed by the item's own option labels; score probabilities keyed by level index '0'..'n-1' - 0-based, unlike ClassOne's), so the vendored typesafe adapter's request body, answer mapping and validations are reused UNCHANGED. or_decisions_r58.OpenRouterDecisionsAdapter rewrites ONLY the URL, because OpenRouter serves the contract at POST /api/alpha/decisions rather than /v1/systemone - its own 400 from chat/completions names that path - and it does so by calling the parent run(), so none of the reviewed mapping is duplicated. The pinned harness/vendor tree was NOT edited; the module is reached by PYTHONPATH. THE 23 LONG ITEMS: on this pool exactly 23 of 1,500 items are 77,857-82,318 tokens and the 24th largest is below 2,048 (run 57's own per-item counts on the identical pool, receipt LONGITEM-TOKENS-R58.json). They are also the 23 items the v1.6.1 cost rule excludes. THIS ROW REFUSES THEM: the context window is 32,768 tokens and all 23 came back as HTTP 422 with the provider's own 'upstream_error: Decision service could not complete the request' - a disclosed refusal, which the harness counts as the system's own rather than as an outage. The invalid set is EXACTLY those 23 and nothing else failed in 1,477 answers. RUN IN TWO PASSES BECAUSE OF A FREE-TIER BURST LIMIT, AND IT DOES NOT MOVE THE SPEED FIGURE: items 1-736 ran back to back at ~1.7 req/s until OpenRouter returned a header-less 429 ('rate_limit_exceeded: Request or spending limit exceeded'), which is run_v16's own stop rule; the row was resumed at 1 req/s and completed. The two passes' medians differ by 0.038 s (0.3624 s vs 0.3246 s over 1,477 answers; 0.3624 vs 0.3104 on the 124 items the Speed axis actually uses), and run_v16 sleeps the spacing BEFORE starting the timer, so the pacing is not in the measurement. Receipt LATENCY-SEGMENTS-mercury-R58.txt. Production API, so NO latency adjustment. Serial, one decision per request, no retry, no batching. add-requests run 58", "adapter_id": null, "measurement_date_status": "run output completion day (UTC)"} Distinct from Leanmcp JevBench and community HF Decision Index. WebMCP feed reportsv1.6.29 while page shell saysboardv1.7.46; preserve exact feed revision and source artifact. Intelligence is chance-corrected/weighted. Calibration is a mixed source index, not ECE. Cost and four-axis composite incorporate an explicitly labeled Mercury2.5 proxy tariff because the original7October run was free; this is historical source methodology, not current Mercury Decide pricing. No paid tariff is inferred. Source-computed values are retained; no Examenos recomputation. Per-type support/validity details are in each record. Distinct from Leanmcp JevBench and community HF Decision Index. WebMCP feed reportsv1.6.29 while page shell saysboardv1.7.46; preserve exact feed revision and source artifact. Intelligence is chance-corrected/weighted. Calibration is a mixed source index, not ECE. Cost and four-axis composite incorporate an explicitly labeled Mercury2.5 proxy tariff because the original7October run was free; this is historical source methodology, not current Mercury Decide pricing. No paid tariff is inferred. Source-computed values are retained; no Examenos recomputation. Per-type support/validity details are in each record. Checked / reported: 2026-10-20 · Samples: 369 Snapshot: 2026-10-20 · Revision: v1.6.1; source_sha256=e15347783b8bedeeddd9017a831608a957f5bf0386f05902c50161ce0468f911 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
Benchmark Heaven JevBench1.6.1 · Calibration · score · top_ece	13.131 pp	Setup & source Benchmark Heaven JevBench1.6.1 · Calibration · score · top_ece Benchmark Heaven v1.6.1; measured_in=v1.6.1; named hosted inception/mercury-decide:free.1500decisions (300public+1200sealed); source identity {"model_pin": "provider-reported measurement model slug: inception/mercury-decide:free", "last_measured_on": "2026-10-07", "endpoint_condition": "measured 2026-10-07 on the live v1.6.0/v1.6.1 pool (inputs/input-selfhosted-1500.jsonl, sha256 901983ae...), COMPLETE: 1,500 of 1,500 dispatched, 1,477 answered, 23 invalid. Measured through the release lane's reviewed run_api_full_v161.py UNCHANGED, on Florian's 5 Oct amendment (hosted systems answer the same full S u P as self-hosted ones); provider inception, proxy openrouter, every sealed item exposure-logged before the first request. Florian work order of 6 Oct 2026 ('put it on JevBench', under https://x.com/airesearch12/status/2107569862486659111). NO MAPPING DECISION OF OURS: Mercury Decide publishes the same /v1/systemone contract as Jev and all three primitives come back in the canonical shape (noul a float; choice a label plus probabilities keyed by the item's own option labels; score probabilities keyed by level index '0'..'n-1' - 0-based, unlike ClassOne's), so the vendored typesafe adapter's request body, answer mapping and validations are reused UNCHANGED. or_decisions_r58.OpenRouterDecisionsAdapter rewrites ONLY the URL, because OpenRouter serves the contract at POST /api/alpha/decisions rather than /v1/systemone - its own 400 from chat/completions names that path - and it does so by calling the parent run(), so none of the reviewed mapping is duplicated. The pinned harness/vendor tree was NOT edited; the module is reached by PYTHONPATH. THE 23 LONG ITEMS: on this pool exactly 23 of 1,500 items are 77,857-82,318 tokens and the 24th largest is below 2,048 (run 57's own per-item counts on the identical pool, receipt LONGITEM-TOKENS-R58.json). They are also the 23 items the v1.6.1 cost rule excludes. THIS ROW REFUSES THEM: the context window is 32,768 tokens and all 23 came back as HTTP 422 with the provider's own 'upstream_error: Decision service could not complete the request' - a disclosed refusal, which the harness counts as the system's own rather than as an outage. The invalid set is EXACTLY those 23 and nothing else failed in 1,477 answers. RUN IN TWO PASSES BECAUSE OF A FREE-TIER BURST LIMIT, AND IT DOES NOT MOVE THE SPEED FIGURE: items 1-736 ran back to back at ~1.7 req/s until OpenRouter returned a header-less 429 ('rate_limit_exceeded: Request or spending limit exceeded'), which is run_v16's own stop rule; the row was resumed at 1 req/s and completed. The two passes' medians differ by 0.038 s (0.3624 s vs 0.3246 s over 1,477 answers; 0.3624 vs 0.3104 on the 124 items the Speed axis actually uses), and run_v16 sleeps the spacing BEFORE starting the timer, so the pacing is not in the measurement. Receipt LATENCY-SEGMENTS-mercury-R58.txt. Production API, so NO latency adjustment. Serial, one decision per request, no retry, no batching. add-requests run 58", "adapter_id": null, "measurement_date_status": "run output completion day (UTC)"} Distinct from Leanmcp JevBench and community HF Decision Index. WebMCP feed reportsv1.6.29 while page shell saysboardv1.7.46; preserve exact feed revision and source artifact. Intelligence is chance-corrected/weighted. Calibration is a mixed source index, not ECE. Cost and four-axis composite incorporate an explicitly labeled Mercury2.5 proxy tariff because the original7October run was free; this is historical source methodology, not current Mercury Decide pricing. No paid tariff is inferred. Source-computed values are retained; no Examenos recomputation. Per-type support/validity details are in each record. Distinct from Leanmcp JevBench and community HF Decision Index. WebMCP feed reportsv1.6.29 while page shell saysboardv1.7.46; preserve exact feed revision and source artifact. Intelligence is chance-corrected/weighted. Calibration is a mixed source index, not ECE. Cost and four-axis composite incorporate an explicitly labeled Mercury2.5 proxy tariff because the original7October run was free; this is historical source methodology, not current Mercury Decide pricing. No paid tariff is inferred. Source-computed values are retained; no Examenos recomputation. Per-type support/validity details are in each record. Checked / reported: 2026-10-20 · Samples: 369 Snapshot: 2026-10-20 · Revision: v1.6.1; source_sha256=e15347783b8bedeeddd9017a831608a957f5bf0386f05902c50161ce0468f911 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
Benchmark Heaven JevBench1.6.1 · Calibration · score · interval90_coverage	87.534%	Setup & source Benchmark Heaven JevBench1.6.1 · Calibration · score · interval90_coverage Benchmark Heaven v1.6.1; measured_in=v1.6.1; named hosted inception/mercury-decide:free.1500decisions (300public+1200sealed); source identity {"model_pin": "provider-reported measurement model slug: inception/mercury-decide:free", "last_measured_on": "2026-10-07", "endpoint_condition": "measured 2026-10-07 on the live v1.6.0/v1.6.1 pool (inputs/input-selfhosted-1500.jsonl, sha256 901983ae...), COMPLETE: 1,500 of 1,500 dispatched, 1,477 answered, 23 invalid. Measured through the release lane's reviewed run_api_full_v161.py UNCHANGED, on Florian's 5 Oct amendment (hosted systems answer the same full S u P as self-hosted ones); provider inception, proxy openrouter, every sealed item exposure-logged before the first request. Florian work order of 6 Oct 2026 ('put it on JevBench', under https://x.com/airesearch12/status/2107569862486659111). NO MAPPING DECISION OF OURS: Mercury Decide publishes the same /v1/systemone contract as Jev and all three primitives come back in the canonical shape (noul a float; choice a label plus probabilities keyed by the item's own option labels; score probabilities keyed by level index '0'..'n-1' - 0-based, unlike ClassOne's), so the vendored typesafe adapter's request body, answer mapping and validations are reused UNCHANGED. or_decisions_r58.OpenRouterDecisionsAdapter rewrites ONLY the URL, because OpenRouter serves the contract at POST /api/alpha/decisions rather than /v1/systemone - its own 400 from chat/completions names that path - and it does so by calling the parent run(), so none of the reviewed mapping is duplicated. The pinned harness/vendor tree was NOT edited; the module is reached by PYTHONPATH. THE 23 LONG ITEMS: on this pool exactly 23 of 1,500 items are 77,857-82,318 tokens and the 24th largest is below 2,048 (run 57's own per-item counts on the identical pool, receipt LONGITEM-TOKENS-R58.json). They are also the 23 items the v1.6.1 cost rule excludes. THIS ROW REFUSES THEM: the context window is 32,768 tokens and all 23 came back as HTTP 422 with the provider's own 'upstream_error: Decision service could not complete the request' - a disclosed refusal, which the harness counts as the system's own rather than as an outage. The invalid set is EXACTLY those 23 and nothing else failed in 1,477 answers. RUN IN TWO PASSES BECAUSE OF A FREE-TIER BURST LIMIT, AND IT DOES NOT MOVE THE SPEED FIGURE: items 1-736 ran back to back at ~1.7 req/s until OpenRouter returned a header-less 429 ('rate_limit_exceeded: Request or spending limit exceeded'), which is run_v16's own stop rule; the row was resumed at 1 req/s and completed. The two passes' medians differ by 0.038 s (0.3624 s vs 0.3246 s over 1,477 answers; 0.3624 vs 0.3104 on the 124 items the Speed axis actually uses), and run_v16 sleeps the spacing BEFORE starting the timer, so the pacing is not in the measurement. Receipt LATENCY-SEGMENTS-mercury-R58.txt. Production API, so NO latency adjustment. Serial, one decision per request, no retry, no batching. add-requests run 58", "adapter_id": null, "measurement_date_status": "run output completion day (UTC)"} Distinct from Leanmcp JevBench and community HF Decision Index. WebMCP feed reportsv1.6.29 while page shell saysboardv1.7.46; preserve exact feed revision and source artifact. Intelligence is chance-corrected/weighted. Calibration is a mixed source index, not ECE. Cost and four-axis composite incorporate an explicitly labeled Mercury2.5 proxy tariff because the original7October run was free; this is historical source methodology, not current Mercury Decide pricing. No paid tariff is inferred. Source-computed values are retained; no Examenos recomputation. Per-type support/validity details are in each record. Distinct from Leanmcp JevBench and community HF Decision Index. WebMCP feed reportsv1.6.29 while page shell saysboardv1.7.46; preserve exact feed revision and source artifact. Intelligence is chance-corrected/weighted. Calibration is a mixed source index, not ECE. Cost and four-axis composite incorporate an explicitly labeled Mercury2.5 proxy tariff because the original7October run was free; this is historical source methodology, not current Mercury Decide pricing. No paid tariff is inferred. Source-computed values are retained; no Examenos recomputation. Per-type support/validity details are in each record. Checked / reported: 2026-10-20 · Samples: 369 Snapshot: 2026-10-20 · Revision: v1.6.1; source_sha256=e15347783b8bedeeddd9017a831608a957f5bf0386f05902c50161ce0468f911 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
Benchmark Heaven JevBench1.6.1 · Calibration · score · interval90_width	14.589%	Setup & source Benchmark Heaven JevBench1.6.1 · Calibration · score · interval90_width Benchmark Heaven v1.6.1; measured_in=v1.6.1; named hosted inception/mercury-decide:free.1500decisions (300public+1200sealed); source identity {"model_pin": "provider-reported measurement model slug: inception/mercury-decide:free", "last_measured_on": "2026-10-07", "endpoint_condition": "measured 2026-10-07 on the live v1.6.0/v1.6.1 pool (inputs/input-selfhosted-1500.jsonl, sha256 901983ae...), COMPLETE: 1,500 of 1,500 dispatched, 1,477 answered, 23 invalid. Measured through the release lane's reviewed run_api_full_v161.py UNCHANGED, on Florian's 5 Oct amendment (hosted systems answer the same full S u P as self-hosted ones); provider inception, proxy openrouter, every sealed item exposure-logged before the first request. Florian work order of 6 Oct 2026 ('put it on JevBench', under https://x.com/airesearch12/status/2107569862486659111). NO MAPPING DECISION OF OURS: Mercury Decide publishes the same /v1/systemone contract as Jev and all three primitives come back in the canonical shape (noul a float; choice a label plus probabilities keyed by the item's own option labels; score probabilities keyed by level index '0'..'n-1' - 0-based, unlike ClassOne's), so the vendored typesafe adapter's request body, answer mapping and validations are reused UNCHANGED. or_decisions_r58.OpenRouterDecisionsAdapter rewrites ONLY the URL, because OpenRouter serves the contract at POST /api/alpha/decisions rather than /v1/systemone - its own 400 from chat/completions names that path - and it does so by calling the parent run(), so none of the reviewed mapping is duplicated. The pinned harness/vendor tree was NOT edited; the module is reached by PYTHONPATH. THE 23 LONG ITEMS: on this pool exactly 23 of 1,500 items are 77,857-82,318 tokens and the 24th largest is below 2,048 (run 57's own per-item counts on the identical pool, receipt LONGITEM-TOKENS-R58.json). They are also the 23 items the v1.6.1 cost rule excludes. THIS ROW REFUSES THEM: the context window is 32,768 tokens and all 23 came back as HTTP 422 with the provider's own 'upstream_error: Decision service could not complete the request' - a disclosed refusal, which the harness counts as the system's own rather than as an outage. The invalid set is EXACTLY those 23 and nothing else failed in 1,477 answers. RUN IN TWO PASSES BECAUSE OF A FREE-TIER BURST LIMIT, AND IT DOES NOT MOVE THE SPEED FIGURE: items 1-736 ran back to back at ~1.7 req/s until OpenRouter returned a header-less 429 ('rate_limit_exceeded: Request or spending limit exceeded'), which is run_v16's own stop rule; the row was resumed at 1 req/s and completed. The two passes' medians differ by 0.038 s (0.3624 s vs 0.3246 s over 1,477 answers; 0.3624 vs 0.3104 on the 124 items the Speed axis actually uses), and run_v16 sleeps the spacing BEFORE starting the timer, so the pacing is not in the measurement. Receipt LATENCY-SEGMENTS-mercury-R58.txt. Production API, so NO latency adjustment. Serial, one decision per request, no retry, no batching. add-requests run 58", "adapter_id": null, "measurement_date_status": "run output completion day (UTC)"} Distinct from Leanmcp JevBench and community HF Decision Index. WebMCP feed reportsv1.6.29 while page shell saysboardv1.7.46; preserve exact feed revision and source artifact. Intelligence is chance-corrected/weighted. Calibration is a mixed source index, not ECE. Cost and four-axis composite incorporate an explicitly labeled Mercury2.5 proxy tariff because the original7October run was free; this is historical source methodology, not current Mercury Decide pricing. No paid tariff is inferred. Source-computed values are retained; no Examenos recomputation. Per-type support/validity details are in each record. Distinct from Leanmcp JevBench and community HF Decision Index. WebMCP feed reportsv1.6.29 while page shell saysboardv1.7.46; preserve exact feed revision and source artifact. Intelligence is chance-corrected/weighted. Calibration is a mixed source index, not ECE. Cost and four-axis composite incorporate an explicitly labeled Mercury2.5 proxy tariff because the original7October run was free; this is historical source methodology, not current Mercury Decide pricing. No paid tariff is inferred. Source-computed values are retained; no Examenos recomputation. Per-type support/validity details are in each record. Checked / reported: 2026-10-20 · Samples: 369 Snapshot: 2026-10-20 · Revision: v1.6.1; source_sha256=e15347783b8bedeeddd9017a831608a957f5bf0386f05902c50161ce0468f911 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
Benchmark Heaven JevBench1.6.1 · Timing · p50_s_raw	0.332 s	Setup & source Benchmark Heaven JevBench1.6.1 · Timing · p50_s_raw Benchmark Heaven v1.6.1; measured_in=v1.6.1; named hosted inception/mercury-decide:free.1500decisions (300public+1200sealed); source identity {"model_pin": "provider-reported measurement model slug: inception/mercury-decide:free", "last_measured_on": "2026-10-07", "endpoint_condition": "measured 2026-10-07 on the live v1.6.0/v1.6.1 pool (inputs/input-selfhosted-1500.jsonl, sha256 901983ae...), COMPLETE: 1,500 of 1,500 dispatched, 1,477 answered, 23 invalid. Measured through the release lane's reviewed run_api_full_v161.py UNCHANGED, on Florian's 5 Oct amendment (hosted systems answer the same full S u P as self-hosted ones); provider inception, proxy openrouter, every sealed item exposure-logged before the first request. Florian work order of 6 Oct 2026 ('put it on JevBench', under https://x.com/airesearch12/status/2107569862486659111). NO MAPPING DECISION OF OURS: Mercury Decide publishes the same /v1/systemone contract as Jev and all three primitives come back in the canonical shape (noul a float; choice a label plus probabilities keyed by the item's own option labels; score probabilities keyed by level index '0'..'n-1' - 0-based, unlike ClassOne's), so the vendored typesafe adapter's request body, answer mapping and validations are reused UNCHANGED. or_decisions_r58.OpenRouterDecisionsAdapter rewrites ONLY the URL, because OpenRouter serves the contract at POST /api/alpha/decisions rather than /v1/systemone - its own 400 from chat/completions names that path - and it does so by calling the parent run(), so none of the reviewed mapping is duplicated. The pinned harness/vendor tree was NOT edited; the module is reached by PYTHONPATH. THE 23 LONG ITEMS: on this pool exactly 23 of 1,500 items are 77,857-82,318 tokens and the 24th largest is below 2,048 (run 57's own per-item counts on the identical pool, receipt LONGITEM-TOKENS-R58.json). They are also the 23 items the v1.6.1 cost rule excludes. THIS ROW REFUSES THEM: the context window is 32,768 tokens and all 23 came back as HTTP 422 with the provider's own 'upstream_error: Decision service could not complete the request' - a disclosed refusal, which the harness counts as the system's own rather than as an outage. The invalid set is EXACTLY those 23 and nothing else failed in 1,477 answers. RUN IN TWO PASSES BECAUSE OF A FREE-TIER BURST LIMIT, AND IT DOES NOT MOVE THE SPEED FIGURE: items 1-736 ran back to back at ~1.7 req/s until OpenRouter returned a header-less 429 ('rate_limit_exceeded: Request or spending limit exceeded'), which is run_v16's own stop rule; the row was resumed at 1 req/s and completed. The two passes' medians differ by 0.038 s (0.3624 s vs 0.3246 s over 1,477 answers; 0.3624 vs 0.3104 on the 124 items the Speed axis actually uses), and run_v16 sleeps the spacing BEFORE starting the timer, so the pacing is not in the measurement. Receipt LATENCY-SEGMENTS-mercury-R58.txt. Production API, so NO latency adjustment. Serial, one decision per request, no retry, no batching. add-requests run 58", "adapter_id": null, "measurement_date_status": "run output completion day (UTC)"} Distinct from Leanmcp JevBench and community HF Decision Index. WebMCP feed reportsv1.6.29 while page shell saysboardv1.7.46; preserve exact feed revision and source artifact. Intelligence is chance-corrected/weighted. Calibration is a mixed source index, not ECE. Cost and four-axis composite incorporate an explicitly labeled Mercury2.5 proxy tariff because the original7October run was free; this is historical source methodology, not current Mercury Decide pricing. No paid tariff is inferred. Source-computed values are retained; no Examenos recomputation. Per-type support/validity details are in each record. Distinct from Leanmcp JevBench and community HF Decision Index. WebMCP feed reportsv1.6.29 while page shell saysboardv1.7.46; preserve exact feed revision and source artifact. Intelligence is chance-corrected/weighted. Calibration is a mixed source index, not ECE. Cost and four-axis composite incorporate an explicitly labeled Mercury2.5 proxy tariff because the original7October run was free; this is historical source methodology, not current Mercury Decide pricing. No paid tariff is inferred. Source-computed values are retained; no Examenos recomputation. Per-type support/validity details are in each record. Checked / reported: 2026-10-20 · Samples: 124 Snapshot: 2026-10-20 · Revision: v1.6.1; source_sha256=e15347783b8bedeeddd9017a831608a957f5bf0386f05902c50161ce0468f911 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
Benchmark Heaven JevBench1.6.1 · Timing · p50_s_adjusted	0.332 s	Setup & source Benchmark Heaven JevBench1.6.1 · Timing · p50_s_adjusted Benchmark Heaven v1.6.1; measured_in=v1.6.1; named hosted inception/mercury-decide:free.1500decisions (300public+1200sealed); source identity {"model_pin": "provider-reported measurement model slug: inception/mercury-decide:free", "last_measured_on": "2026-10-07", "endpoint_condition": "measured 2026-10-07 on the live v1.6.0/v1.6.1 pool (inputs/input-selfhosted-1500.jsonl, sha256 901983ae...), COMPLETE: 1,500 of 1,500 dispatched, 1,477 answered, 23 invalid. Measured through the release lane's reviewed run_api_full_v161.py UNCHANGED, on Florian's 5 Oct amendment (hosted systems answer the same full S u P as self-hosted ones); provider inception, proxy openrouter, every sealed item exposure-logged before the first request. Florian work order of 6 Oct 2026 ('put it on JevBench', under https://x.com/airesearch12/status/2107569862486659111). NO MAPPING DECISION OF OURS: Mercury Decide publishes the same /v1/systemone contract as Jev and all three primitives come back in the canonical shape (noul a float; choice a label plus probabilities keyed by the item's own option labels; score probabilities keyed by level index '0'..'n-1' - 0-based, unlike ClassOne's), so the vendored typesafe adapter's request body, answer mapping and validations are reused UNCHANGED. or_decisions_r58.OpenRouterDecisionsAdapter rewrites ONLY the URL, because OpenRouter serves the contract at POST /api/alpha/decisions rather than /v1/systemone - its own 400 from chat/completions names that path - and it does so by calling the parent run(), so none of the reviewed mapping is duplicated. The pinned harness/vendor tree was NOT edited; the module is reached by PYTHONPATH. THE 23 LONG ITEMS: on this pool exactly 23 of 1,500 items are 77,857-82,318 tokens and the 24th largest is below 2,048 (run 57's own per-item counts on the identical pool, receipt LONGITEM-TOKENS-R58.json). They are also the 23 items the v1.6.1 cost rule excludes. THIS ROW REFUSES THEM: the context window is 32,768 tokens and all 23 came back as HTTP 422 with the provider's own 'upstream_error: Decision service could not complete the request' - a disclosed refusal, which the harness counts as the system's own rather than as an outage. The invalid set is EXACTLY those 23 and nothing else failed in 1,477 answers. RUN IN TWO PASSES BECAUSE OF A FREE-TIER BURST LIMIT, AND IT DOES NOT MOVE THE SPEED FIGURE: items 1-736 ran back to back at ~1.7 req/s until OpenRouter returned a header-less 429 ('rate_limit_exceeded: Request or spending limit exceeded'), which is run_v16's own stop rule; the row was resumed at 1 req/s and completed. The two passes' medians differ by 0.038 s (0.3624 s vs 0.3246 s over 1,477 answers; 0.3624 vs 0.3104 on the 124 items the Speed axis actually uses), and run_v16 sleeps the spacing BEFORE starting the timer, so the pacing is not in the measurement. Receipt LATENCY-SEGMENTS-mercury-R58.txt. Production API, so NO latency adjustment. Serial, one decision per request, no retry, no batching. add-requests run 58", "adapter_id": null, "measurement_date_status": "run output completion day (UTC)"} Distinct from Leanmcp JevBench and community HF Decision Index. WebMCP feed reportsv1.6.29 while page shell saysboardv1.7.46; preserve exact feed revision and source artifact. Intelligence is chance-corrected/weighted. Calibration is a mixed source index, not ECE. Cost and four-axis composite incorporate an explicitly labeled Mercury2.5 proxy tariff because the original7October run was free; this is historical source methodology, not current Mercury Decide pricing. No paid tariff is inferred. Source-computed values are retained; no Examenos recomputation. Per-type support/validity details are in each record. Distinct from Leanmcp JevBench and community HF Decision Index. WebMCP feed reportsv1.6.29 while page shell saysboardv1.7.46; preserve exact feed revision and source artifact. Intelligence is chance-corrected/weighted. Calibration is a mixed source index, not ECE. Cost and four-axis composite incorporate an explicitly labeled Mercury2.5 proxy tariff because the original7October run was free; this is historical source methodology, not current Mercury Decide pricing. No paid tariff is inferred. Source-computed values are retained; no Examenos recomputation. Per-type support/validity details are in each record. Checked / reported: 2026-10-20 · Samples: 124 Snapshot: 2026-10-20 · Revision: v1.6.1; source_sha256=e15347783b8bedeeddd9017a831608a957f5bf0386f05902c50161ce0468f911 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
Benchmark Heaven JevBench1.6.1 · Timing · p95_s_raw	0.853 s	Setup & source Benchmark Heaven JevBench1.6.1 · Timing · p95_s_raw Benchmark Heaven v1.6.1; measured_in=v1.6.1; named hosted inception/mercury-decide:free.1500decisions (300public+1200sealed); source identity {"model_pin": "provider-reported measurement model slug: inception/mercury-decide:free", "last_measured_on": "2026-10-07", "endpoint_condition": "measured 2026-10-07 on the live v1.6.0/v1.6.1 pool (inputs/input-selfhosted-1500.jsonl, sha256 901983ae...), COMPLETE: 1,500 of 1,500 dispatched, 1,477 answered, 23 invalid. Measured through the release lane's reviewed run_api_full_v161.py UNCHANGED, on Florian's 5 Oct amendment (hosted systems answer the same full S u P as self-hosted ones); provider inception, proxy openrouter, every sealed item exposure-logged before the first request. Florian work order of 6 Oct 2026 ('put it on JevBench', under https://x.com/airesearch12/status/2107569862486659111). NO MAPPING DECISION OF OURS: Mercury Decide publishes the same /v1/systemone contract as Jev and all three primitives come back in the canonical shape (noul a float; choice a label plus probabilities keyed by the item's own option labels; score probabilities keyed by level index '0'..'n-1' - 0-based, unlike ClassOne's), so the vendored typesafe adapter's request body, answer mapping and validations are reused UNCHANGED. or_decisions_r58.OpenRouterDecisionsAdapter rewrites ONLY the URL, because OpenRouter serves the contract at POST /api/alpha/decisions rather than /v1/systemone - its own 400 from chat/completions names that path - and it does so by calling the parent run(), so none of the reviewed mapping is duplicated. The pinned harness/vendor tree was NOT edited; the module is reached by PYTHONPATH. THE 23 LONG ITEMS: on this pool exactly 23 of 1,500 items are 77,857-82,318 tokens and the 24th largest is below 2,048 (run 57's own per-item counts on the identical pool, receipt LONGITEM-TOKENS-R58.json). They are also the 23 items the v1.6.1 cost rule excludes. THIS ROW REFUSES THEM: the context window is 32,768 tokens and all 23 came back as HTTP 422 with the provider's own 'upstream_error: Decision service could not complete the request' - a disclosed refusal, which the harness counts as the system's own rather than as an outage. The invalid set is EXACTLY those 23 and nothing else failed in 1,477 answers. RUN IN TWO PASSES BECAUSE OF A FREE-TIER BURST LIMIT, AND IT DOES NOT MOVE THE SPEED FIGURE: items 1-736 ran back to back at ~1.7 req/s until OpenRouter returned a header-less 429 ('rate_limit_exceeded: Request or spending limit exceeded'), which is run_v16's own stop rule; the row was resumed at 1 req/s and completed. The two passes' medians differ by 0.038 s (0.3624 s vs 0.3246 s over 1,477 answers; 0.3624 vs 0.3104 on the 124 items the Speed axis actually uses), and run_v16 sleeps the spacing BEFORE starting the timer, so the pacing is not in the measurement. Receipt LATENCY-SEGMENTS-mercury-R58.txt. Production API, so NO latency adjustment. Serial, one decision per request, no retry, no batching. add-requests run 58", "adapter_id": null, "measurement_date_status": "run output completion day (UTC)"} Distinct from Leanmcp JevBench and community HF Decision Index. WebMCP feed reportsv1.6.29 while page shell saysboardv1.7.46; preserve exact feed revision and source artifact. Intelligence is chance-corrected/weighted. Calibration is a mixed source index, not ECE. Cost and four-axis composite incorporate an explicitly labeled Mercury2.5 proxy tariff because the original7October run was free; this is historical source methodology, not current Mercury Decide pricing. No paid tariff is inferred. Source-computed values are retained; no Examenos recomputation. Per-type support/validity details are in each record. Distinct from Leanmcp JevBench and community HF Decision Index. WebMCP feed reportsv1.6.29 while page shell saysboardv1.7.46; preserve exact feed revision and source artifact. Intelligence is chance-corrected/weighted. Calibration is a mixed source index, not ECE. Cost and four-axis composite incorporate an explicitly labeled Mercury2.5 proxy tariff because the original7October run was free; this is historical source methodology, not current Mercury Decide pricing. No paid tariff is inferred. Source-computed values are retained; no Examenos recomputation. Per-type support/validity details are in each record. Checked / reported: 2026-10-20 · Samples: 124 Snapshot: 2026-10-20 · Revision: v1.6.1; source_sha256=e15347783b8bedeeddd9017a831608a957f5bf0386f05902c50161ce0468f911 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
Benchmark Heaven JevBench1.6.1 · Timing · p95_s_adjusted	0.853 s	Setup & source Benchmark Heaven JevBench1.6.1 · Timing · p95_s_adjusted Benchmark Heaven v1.6.1; measured_in=v1.6.1; named hosted inception/mercury-decide:free.1500decisions (300public+1200sealed); source identity {"model_pin": "provider-reported measurement model slug: inception/mercury-decide:free", "last_measured_on": "2026-10-07", "endpoint_condition": "measured 2026-10-07 on the live v1.6.0/v1.6.1 pool (inputs/input-selfhosted-1500.jsonl, sha256 901983ae...), COMPLETE: 1,500 of 1,500 dispatched, 1,477 answered, 23 invalid. Measured through the release lane's reviewed run_api_full_v161.py UNCHANGED, on Florian's 5 Oct amendment (hosted systems answer the same full S u P as self-hosted ones); provider inception, proxy openrouter, every sealed item exposure-logged before the first request. Florian work order of 6 Oct 2026 ('put it on JevBench', under https://x.com/airesearch12/status/2107569862486659111). NO MAPPING DECISION OF OURS: Mercury Decide publishes the same /v1/systemone contract as Jev and all three primitives come back in the canonical shape (noul a float; choice a label plus probabilities keyed by the item's own option labels; score probabilities keyed by level index '0'..'n-1' - 0-based, unlike ClassOne's), so the vendored typesafe adapter's request body, answer mapping and validations are reused UNCHANGED. or_decisions_r58.OpenRouterDecisionsAdapter rewrites ONLY the URL, because OpenRouter serves the contract at POST /api/alpha/decisions rather than /v1/systemone - its own 400 from chat/completions names that path - and it does so by calling the parent run(), so none of the reviewed mapping is duplicated. The pinned harness/vendor tree was NOT edited; the module is reached by PYTHONPATH. THE 23 LONG ITEMS: on this pool exactly 23 of 1,500 items are 77,857-82,318 tokens and the 24th largest is below 2,048 (run 57's own per-item counts on the identical pool, receipt LONGITEM-TOKENS-R58.json). They are also the 23 items the v1.6.1 cost rule excludes. THIS ROW REFUSES THEM: the context window is 32,768 tokens and all 23 came back as HTTP 422 with the provider's own 'upstream_error: Decision service could not complete the request' - a disclosed refusal, which the harness counts as the system's own rather than as an outage. The invalid set is EXACTLY those 23 and nothing else failed in 1,477 answers. RUN IN TWO PASSES BECAUSE OF A FREE-TIER BURST LIMIT, AND IT DOES NOT MOVE THE SPEED FIGURE: items 1-736 ran back to back at ~1.7 req/s until OpenRouter returned a header-less 429 ('rate_limit_exceeded: Request or spending limit exceeded'), which is run_v16's own stop rule; the row was resumed at 1 req/s and completed. The two passes' medians differ by 0.038 s (0.3624 s vs 0.3246 s over 1,477 answers; 0.3624 vs 0.3104 on the 124 items the Speed axis actually uses), and run_v16 sleeps the spacing BEFORE starting the timer, so the pacing is not in the measurement. Receipt LATENCY-SEGMENTS-mercury-R58.txt. Production API, so NO latency adjustment. Serial, one decision per request, no retry, no batching. add-requests run 58", "adapter_id": null, "measurement_date_status": "run output completion day (UTC)"} Distinct from Leanmcp JevBench and community HF Decision Index. WebMCP feed reportsv1.6.29 while page shell saysboardv1.7.46; preserve exact feed revision and source artifact. Intelligence is chance-corrected/weighted. Calibration is a mixed source index, not ECE. Cost and four-axis composite incorporate an explicitly labeled Mercury2.5 proxy tariff because the original7October run was free; this is historical source methodology, not current Mercury Decide pricing. No paid tariff is inferred. Source-computed values are retained; no Examenos recomputation. Per-type support/validity details are in each record. Distinct from Leanmcp JevBench and community HF Decision Index. WebMCP feed reportsv1.6.29 while page shell saysboardv1.7.46; preserve exact feed revision and source artifact. Intelligence is chance-corrected/weighted. Calibration is a mixed source index, not ECE. Cost and four-axis composite incorporate an explicitly labeled Mercury2.5 proxy tariff because the original7October run was free; this is historical source methodology, not current Mercury Decide pricing. No paid tariff is inferred. Source-computed values are retained; no Examenos recomputation. Per-type support/validity details are in each record. Checked / reported: 2026-10-20 · Samples: 124 Snapshot: 2026-10-20 · Revision: v1.6.1; source_sha256=e15347783b8bedeeddd9017a831608a957f5bf0386f05902c50161ce0468f911 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
Benchmark Heaven JevBench1.6.1 · Historical proxy estimated cost	0.01836 USD/1k decisions	Setup & source Benchmark Heaven JevBench1.6.1 · Historical proxy estimated cost Benchmark Heaven v1.6.1; measured_in=v1.6.1; named hosted inception/mercury-decide:free.1500decisions (300public+1200sealed); source identity {"model_pin": "provider-reported measurement model slug: inception/mercury-decide:free", "last_measured_on": "2026-10-07", "endpoint_condition": "measured 2026-10-07 on the live v1.6.0/v1.6.1 pool (inputs/input-selfhosted-1500.jsonl, sha256 901983ae...), COMPLETE: 1,500 of 1,500 dispatched, 1,477 answered, 23 invalid. Measured through the release lane's reviewed run_api_full_v161.py UNCHANGED, on Florian's 5 Oct amendment (hosted systems answer the same full S u P as self-hosted ones); provider inception, proxy openrouter, every sealed item exposure-logged before the first request. Florian work order of 6 Oct 2026 ('put it on JevBench', under https://x.com/airesearch12/status/2107569862486659111). NO MAPPING DECISION OF OURS: Mercury Decide publishes the same /v1/systemone contract as Jev and all three primitives come back in the canonical shape (noul a float; choice a label plus probabilities keyed by the item's own option labels; score probabilities keyed by level index '0'..'n-1' - 0-based, unlike ClassOne's), so the vendored typesafe adapter's request body, answer mapping and validations are reused UNCHANGED. or_decisions_r58.OpenRouterDecisionsAdapter rewrites ONLY the URL, because OpenRouter serves the contract at POST /api/alpha/decisions rather than /v1/systemone - its own 400 from chat/completions names that path - and it does so by calling the parent run(), so none of the reviewed mapping is duplicated. The pinned harness/vendor tree was NOT edited; the module is reached by PYTHONPATH. THE 23 LONG ITEMS: on this pool exactly 23 of 1,500 items are 77,857-82,318 tokens and the 24th largest is below 2,048 (run 57's own per-item counts on the identical pool, receipt LONGITEM-TOKENS-R58.json). They are also the 23 items the v1.6.1 cost rule excludes. THIS ROW REFUSES THEM: the context window is 32,768 tokens and all 23 came back as HTTP 422 with the provider's own 'upstream_error: Decision service could not complete the request' - a disclosed refusal, which the harness counts as the system's own rather than as an outage. The invalid set is EXACTLY those 23 and nothing else failed in 1,477 answers. RUN IN TWO PASSES BECAUSE OF A FREE-TIER BURST LIMIT, AND IT DOES NOT MOVE THE SPEED FIGURE: items 1-736 ran back to back at ~1.7 req/s until OpenRouter returned a header-less 429 ('rate_limit_exceeded: Request or spending limit exceeded'), which is run_v16's own stop rule; the row was resumed at 1 req/s and completed. The two passes' medians differ by 0.038 s (0.3624 s vs 0.3246 s over 1,477 answers; 0.3624 vs 0.3104 on the 124 items the Speed axis actually uses), and run_v16 sleeps the spacing BEFORE starting the timer, so the pacing is not in the measurement. Receipt LATENCY-SEGMENTS-mercury-R58.txt. Production API, so NO latency adjustment. Serial, one decision per request, no retry, no batching. add-requests run 58", "adapter_id": null, "measurement_date_status": "run output completion day (UTC)"} Distinct from Leanmcp JevBench and community HF Decision Index. WebMCP feed reportsv1.6.29 while page shell saysboardv1.7.46; preserve exact feed revision and source artifact. Intelligence is chance-corrected/weighted. Calibration is a mixed source index, not ECE. Cost and four-axis composite incorporate an explicitly labeled Mercury2.5 proxy tariff because the original7October run was free; this is historical source methodology, not current Mercury Decide pricing. No paid tariff is inferred. Source-computed values are retained; no Examenos recomputation. Per-type support/validity details are in each record. Distinct from Leanmcp JevBench and community HF Decision Index. WebMCP feed reportsv1.6.29 while page shell saysboardv1.7.46; preserve exact feed revision and source artifact. Intelligence is chance-corrected/weighted. Calibration is a mixed source index, not ECE. Cost and four-axis composite incorporate an explicitly labeled Mercury2.5 proxy tariff because the original7October run was free; this is historical source methodology, not current Mercury Decide pricing. No paid tariff is inferred. Source-computed values are retained; no Examenos recomputation. Per-type support/validity details are in each record. Checked / reported: 2026-10-20 · Samples: 1500 Snapshot: 2026-10-20 · Revision: v1.6.1; source_sha256=e15347783b8bedeeddd9017a831608a957f5bf0386f05902c50161ce0468f911 · Published summary observations; upstream task/result terms retained Result source · Benchmark source

Mercury Decide · Evaluator-published · etsabary development study
Evaluation	Result	Evidence
Decision reference DEVELOPMENT1342 · All-case accuracy	75.782%	Setup & source Decision reference DEVELOPMENT1342 · All-case accuracy Source consolidated development overview19October2026; Mercury Decide direct native Decision route through OpenRouter, returned inception/mercury-decide-20260930 on1342calls; Inception provider.1136items18October+206items19October (sourceToronto dates). First-attempt measurements, failed deliveries remain failures. NOT a held-out scored release.37author-development cases permanently excluded from release diagnostics; dependent variants/correlated routes and unbalanced31families. Six new domains have only8questions each. No independent-trial inference or broad reliability claim. Mixed original dates/providers/request loads, not synchronized reruns. Median/p95 excludes retry/queue waits. Captured charges/prices are dated workload observations, not token tariffs. NOT a held-out scored release.37author-development cases permanently excluded from release diagnostics; dependent variants/correlated routes and unbalanced31families. Six new domains have only8questions each. No independent-trial inference or broad reliability claim. Mixed original dates/providers/request loads, not synchronized reruns. Median/p95 excludes retry/queue waits. Captured charges/prices are dated workload observations, not token tariffs. Checked / reported: 2026-10-20 · Samples: 1342 Snapshot: 2026-10-19 · Revision: 419978f2753b4e3bb70a8be9aca7fe0c9e59bc9f · Published summary observations; upstream task/result terms retained Result source · Benchmark source
Decision reference DEVELOPMENT1342 · Macro-family accuracy	73.344%	Setup & source Decision reference DEVELOPMENT1342 · Macro-family accuracy Source consolidated development overview19October2026; Mercury Decide direct native Decision route through OpenRouter, returned inception/mercury-decide-20260930 on1342calls; Inception provider.1136items18October+206items19October (sourceToronto dates). First-attempt measurements, failed deliveries remain failures. NOT a held-out scored release.37author-development cases permanently excluded from release diagnostics; dependent variants/correlated routes and unbalanced31families. Six new domains have only8questions each. No independent-trial inference or broad reliability claim. Mixed original dates/providers/request loads, not synchronized reruns. Median/p95 excludes retry/queue waits. Captured charges/prices are dated workload observations, not token tariffs. NOT a held-out scored release.37author-development cases permanently excluded from release diagnostics; dependent variants/correlated routes and unbalanced31families. Six new domains have only8questions each. No independent-trial inference or broad reliability claim. Mixed original dates/providers/request loads, not synchronized reruns. Median/p95 excludes retry/queue waits. Captured charges/prices are dated workload observations, not token tariffs. Checked / reported: 2026-10-20 · Samples: 1342 Snapshot: 2026-10-19 · Revision: 419978f2753b4e3bb70a8be9aca7fe0c9e59bc9f · Published summary observations; upstream task/result terms retained Result source · Benchmark source
Decision reference DEVELOPMENT1342 · median_ms	238.431 ms	Setup & source Decision reference DEVELOPMENT1342 · median_ms Source consolidated development overview19October2026; Mercury Decide direct native Decision route through OpenRouter, returned inception/mercury-decide-20260930 on1342calls; Inception provider.1136items18October+206items19October (sourceToronto dates). First-attempt measurements, failed deliveries remain failures. NOT a held-out scored release.37author-development cases permanently excluded from release diagnostics; dependent variants/correlated routes and unbalanced31families. Six new domains have only8questions each. No independent-trial inference or broad reliability claim. Mixed original dates/providers/request loads, not synchronized reruns. Median/p95 excludes retry/queue waits. Captured charges/prices are dated workload observations, not token tariffs. NOT a held-out scored release.37author-development cases permanently excluded from release diagnostics; dependent variants/correlated routes and unbalanced31families. Six new domains have only8questions each. No independent-trial inference or broad reliability claim. Mixed original dates/providers/request loads, not synchronized reruns. Median/p95 excludes retry/queue waits. Captured charges/prices are dated workload observations, not token tariffs. Checked / reported: 2026-10-20 · Samples: 1342 Snapshot: 2026-10-19 · Revision: 419978f2753b4e3bb70a8be9aca7fe0c9e59bc9f · Published summary observations; upstream task/result terms retained Result source · Benchmark source
Decision reference DEVELOPMENT1342 · p95_ms	388.373 ms	Setup & source Decision reference DEVELOPMENT1342 · p95_ms Source consolidated development overview19October2026; Mercury Decide direct native Decision route through OpenRouter, returned inception/mercury-decide-20260930 on1342calls; Inception provider.1136items18October+206items19October (sourceToronto dates). First-attempt measurements, failed deliveries remain failures. NOT a held-out scored release.37author-development cases permanently excluded from release diagnostics; dependent variants/correlated routes and unbalanced31families. Six new domains have only8questions each. No independent-trial inference or broad reliability claim. Mixed original dates/providers/request loads, not synchronized reruns. Median/p95 excludes retry/queue waits. Captured charges/prices are dated workload observations, not token tariffs. NOT a held-out scored release.37author-development cases permanently excluded from release diagnostics; dependent variants/correlated routes and unbalanced31families. Six new domains have only8questions each. No independent-trial inference or broad reliability claim. Mixed original dates/providers/request loads, not synchronized reruns. Median/p95 excludes retry/queue waits. Captured charges/prices are dated workload observations, not token tariffs. Checked / reported: 2026-10-20 · Samples: 1342 Snapshot: 2026-10-19 · Revision: 419978f2753b4e3bb70a8be9aca7fe0c9e59bc9f · Published summary observations; upstream task/result terms retained Result source · Benchmark source
Decision reference DEVELOPMENT1342 · Captured total cost	0.009334 USD/run	Setup & source Decision reference DEVELOPMENT1342 · Captured total cost Source consolidated development overview19October2026; Mercury Decide direct native Decision route through OpenRouter, returned inception/mercury-decide-20260930 on1342calls; Inception provider.1136items18October+206items19October (sourceToronto dates). First-attempt measurements, failed deliveries remain failures. NOT a held-out scored release.37author-development cases permanently excluded from release diagnostics; dependent variants/correlated routes and unbalanced31families. Six new domains have only8questions each. No independent-trial inference or broad reliability claim. Mixed original dates/providers/request loads, not synchronized reruns. Median/p95 excludes retry/queue waits. Captured charges/prices are dated workload observations, not token tariffs. NOT a held-out scored release.37author-development cases permanently excluded from release diagnostics; dependent variants/correlated routes and unbalanced31families. Six new domains have only8questions each. No independent-trial inference or broad reliability claim. Mixed original dates/providers/request loads, not synchronized reruns. Median/p95 excludes retry/queue waits. Captured charges/prices are dated workload observations, not token tariffs. Checked / reported: 2026-10-20 · Samples: 1342 Snapshot: 2026-10-19 · Revision: 419978f2753b4e3bb70a8be9aca7fe0c9e59bc9f · Published summary observations; upstream task/result terms retained Result source · Benchmark source
Decision reference DEVELOPMENT1342 · Captured cost per1000	0.006956 USD/1k decisions	Setup & source Decision reference DEVELOPMENT1342 · Captured cost per1000 Source consolidated development overview19October2026; Mercury Decide direct native Decision route through OpenRouter, returned inception/mercury-decide-20260930 on1342calls; Inception provider.1136items18October+206items19October (sourceToronto dates). First-attempt measurements, failed deliveries remain failures. NOT a held-out scored release.37author-development cases permanently excluded from release diagnostics; dependent variants/correlated routes and unbalanced31families. Six new domains have only8questions each. No independent-trial inference or broad reliability claim. Mixed original dates/providers/request loads, not synchronized reruns. Median/p95 excludes retry/queue waits. Captured charges/prices are dated workload observations, not token tariffs. NOT a held-out scored release.37author-development cases permanently excluded from release diagnostics; dependent variants/correlated routes and unbalanced31families. Six new domains have only8questions each. No independent-trial inference or broad reliability claim. Mixed original dates/providers/request loads, not synchronized reruns. Median/p95 excludes retry/queue waits. Captured charges/prices are dated workload observations, not token tariffs. Checked / reported: 2026-10-20 · Samples: 1342 Snapshot: 2026-10-19 · Revision: 419978f2753b4e3bb70a8be9aca7fe0c9e59bc9f · Published summary observations; upstream task/result terms retained Result source · Benchmark source
Decision reference DEVELOPMENT1342 · F01 · Lookup and filtering	88.372%	Setup & source Decision reference DEVELOPMENT1342 · F01 · Lookup and filtering Source consolidated development overview19October2026; Mercury Decide direct native Decision route through OpenRouter, returned inception/mercury-decide-20260930 on1342calls; Inception provider.1136items18October+206items19October (sourceToronto dates). First-attempt measurements, failed deliveries remain failures. NOT a held-out scored release.37author-development cases permanently excluded from release diagnostics; dependent variants/correlated routes and unbalanced31families. Six new domains have only8questions each. No independent-trial inference or broad reliability claim. Mixed original dates/providers/request loads, not synchronized reruns. Median/p95 excludes retry/queue waits. Captured charges/prices are dated workload observations, not token tariffs. NOT a held-out scored release.37author-development cases permanently excluded from release diagnostics; dependent variants/correlated routes and unbalanced31families. Six new domains have only8questions each. No independent-trial inference or broad reliability claim. Mixed original dates/providers/request loads, not synchronized reruns. Median/p95 excludes retry/queue waits. Captured charges/prices are dated workload observations, not token tariffs. Checked / reported: 2026-10-20 · Samples: 43 Snapshot: 2026-10-19 · Revision: 419978f2753b4e3bb70a8be9aca7fe0c9e59bc9f · Published summary observations; upstream task/result terms retained Result source · Benchmark source
Decision reference DEVELOPMENT1342 · F02 · Forward conditionals	93.478%	Setup & source Decision reference DEVELOPMENT1342 · F02 · Forward conditionals Source consolidated development overview19October2026; Mercury Decide direct native Decision route through OpenRouter, returned inception/mercury-decide-20260930 on1342calls; Inception provider.1136items18October+206items19October (sourceToronto dates). First-attempt measurements, failed deliveries remain failures. NOT a held-out scored release.37author-development cases permanently excluded from release diagnostics; dependent variants/correlated routes and unbalanced31families. Six new domains have only8questions each. No independent-trial inference or broad reliability claim. Mixed original dates/providers/request loads, not synchronized reruns. Median/p95 excludes retry/queue waits. Captured charges/prices are dated workload observations, not token tariffs. NOT a held-out scored release.37author-development cases permanently excluded from release diagnostics; dependent variants/correlated routes and unbalanced31families. Six new domains have only8questions each. No independent-trial inference or broad reliability claim. Mixed original dates/providers/request loads, not synchronized reruns. Median/p95 excludes retry/queue waits. Captured charges/prices are dated workload observations, not token tariffs. Checked / reported: 2026-10-20 · Samples: 46 Snapshot: 2026-10-19 · Revision: 419978f2753b4e3bb70a8be9aca7fe0c9e59bc9f · Published summary observations; upstream task/result terms retained Result source · Benchmark source
Decision reference DEVELOPMENT1342 · F03 · Necessary conditions	53.846%	Setup & source Decision reference DEVELOPMENT1342 · F03 · Necessary conditions Source consolidated development overview19October2026; Mercury Decide direct native Decision route through OpenRouter, returned inception/mercury-decide-20260930 on1342calls; Inception provider.1136items18October+206items19October (sourceToronto dates). First-attempt measurements, failed deliveries remain failures. NOT a held-out scored release.37author-development cases permanently excluded from release diagnostics; dependent variants/correlated routes and unbalanced31families. Six new domains have only8questions each. No independent-trial inference or broad reliability claim. Mixed original dates/providers/request loads, not synchronized reruns. Median/p95 excludes retry/queue waits. Captured charges/prices are dated workload observations, not token tariffs. NOT a held-out scored release.37author-development cases permanently excluded from release diagnostics; dependent variants/correlated routes and unbalanced31families. Six new domains have only8questions each. No independent-trial inference or broad reliability claim. Mixed original dates/providers/request loads, not synchronized reruns. Median/p95 excludes retry/queue waits. Captured charges/prices are dated workload observations, not token tariffs. Checked / reported: 2026-10-20 · Samples: 52 Snapshot: 2026-10-19 · Revision: 419978f2753b4e3bb70a8be9aca7fe0c9e59bc9f · Published summary observations; upstream task/result terms retained Result source · Benchmark source
Decision reference DEVELOPMENT1342 · F04 · Boolean scope	90.741%	Setup & source Decision reference DEVELOPMENT1342 · F04 · Boolean scope Source consolidated development overview19October2026; Mercury Decide direct native Decision route through OpenRouter, returned inception/mercury-decide-20260930 on1342calls; Inception provider.1136items18October+206items19October (sourceToronto dates). First-attempt measurements, failed deliveries remain failures. NOT a held-out scored release.37author-development cases permanently excluded from release diagnostics; dependent variants/correlated routes and unbalanced31families. Six new domains have only8questions each. No independent-trial inference or broad reliability claim. Mixed original dates/providers/request loads, not synchronized reruns. Median/p95 excludes retry/queue waits. Captured charges/prices are dated workload observations, not token tariffs. NOT a held-out scored release.37author-development cases permanently excluded from release diagnostics; dependent variants/correlated routes and unbalanced31families. Six new domains have only8questions each. No independent-trial inference or broad reliability claim. Mixed original dates/providers/request loads, not synchronized reruns. Median/p95 excludes retry/queue waits. Captured charges/prices are dated workload observations, not token tariffs. Checked / reported: 2026-10-20 · Samples: 54 Snapshot: 2026-10-19 · Revision: 419978f2753b4e3bb70a8be9aca7fe0c9e59bc9f · Published summary observations; upstream task/result terms retained Result source · Benchmark source
Decision reference DEVELOPMENT1342 · F05 · Quantifiers and sets	93.878%	Setup & source Decision reference DEVELOPMENT1342 · F05 · Quantifiers and sets Source consolidated development overview19October2026; Mercury Decide direct native Decision route through OpenRouter, returned inception/mercury-decide-20260930 on1342calls; Inception provider.1136items18October+206items19October (sourceToronto dates). First-attempt measurements, failed deliveries remain failures. NOT a held-out scored release.37author-development cases permanently excluded from release diagnostics; dependent variants/correlated routes and unbalanced31families. Six new domains have only8questions each. No independent-trial inference or broad reliability claim. Mixed original dates/providers/request loads, not synchronized reruns. Median/p95 excludes retry/queue waits. Captured charges/prices are dated workload observations, not token tariffs. NOT a held-out scored release.37author-development cases permanently excluded from release diagnostics; dependent variants/correlated routes and unbalanced31families. Six new domains have only8questions each. No independent-trial inference or broad reliability claim. Mixed original dates/providers/request loads, not synchronized reruns. Median/p95 excludes retry/queue waits. Captured charges/prices are dated workload observations, not token tariffs. Checked / reported: 2026-10-20 · Samples: 49 Snapshot: 2026-10-19 · Revision: 419978f2753b4e3bb70a8be9aca7fe0c9e59bc9f · Published summary observations; upstream task/result terms retained Result source · Benchmark source
Decision reference DEVELOPMENT1342 · F06 · Comparisons	92.500%	Setup & source Decision reference DEVELOPMENT1342 · F06 · Comparisons Source consolidated development overview19October2026; Mercury Decide direct native Decision route through OpenRouter, returned inception/mercury-decide-20260930 on1342calls; Inception provider.1136items18October+206items19October (sourceToronto dates). First-attempt measurements, failed deliveries remain failures. NOT a held-out scored release.37author-development cases permanently excluded from release diagnostics; dependent variants/correlated routes and unbalanced31families. Six new domains have only8questions each. No independent-trial inference or broad reliability claim. Mixed original dates/providers/request loads, not synchronized reruns. Median/p95 excludes retry/queue waits. Captured charges/prices are dated workload observations, not token tariffs. NOT a held-out scored release.37author-development cases permanently excluded from release diagnostics; dependent variants/correlated routes and unbalanced31families. Six new domains have only8questions each. No independent-trial inference or broad reliability claim. Mixed original dates/providers/request loads, not synchronized reruns. Median/p95 excludes retry/queue waits. Captured charges/prices are dated workload observations, not token tariffs. Checked / reported: 2026-10-20 · Samples: 40 Snapshot: 2026-10-19 · Revision: 419978f2753b4e3bb70a8be9aca7fe0c9e59bc9f · Published summary observations; upstream task/result terms retained Result source · Benchmark source
Decision reference DEVELOPMENT1342 · F07 · Scheduling	65.789%	Setup & source Decision reference DEVELOPMENT1342 · F07 · Scheduling Source consolidated development overview19October2026; Mercury Decide direct native Decision route through OpenRouter, returned inception/mercury-decide-20260930 on1342calls; Inception provider.1136items18October+206items19October (sourceToronto dates). First-attempt measurements, failed deliveries remain failures. NOT a held-out scored release.37author-development cases permanently excluded from release diagnostics; dependent variants/correlated routes and unbalanced31families. Six new domains have only8questions each. No independent-trial inference or broad reliability claim. Mixed original dates/providers/request loads, not synchronized reruns. Median/p95 excludes retry/queue waits. Captured charges/prices are dated workload observations, not token tariffs. NOT a held-out scored release.37author-development cases permanently excluded from release diagnostics; dependent variants/correlated routes and unbalanced31families. Six new domains have only8questions each. No independent-trial inference or broad reliability claim. Mixed original dates/providers/request loads, not synchronized reruns. Median/p95 excludes retry/queue waits. Captured charges/prices are dated workload observations, not token tariffs. Checked / reported: 2026-10-20 · Samples: 76 Snapshot: 2026-10-19 · Revision: 419978f2753b4e3bb70a8be9aca7fe0c9e59bc9f · Published summary observations; upstream task/result terms retained Result source · Benchmark source
Decision reference DEVELOPMENT1342 · F08 · One-to-one assignments	77.612%	Setup & source Decision reference DEVELOPMENT1342 · F08 · One-to-one assignments Source consolidated development overview19October2026; Mercury Decide direct native Decision route through OpenRouter, returned inception/mercury-decide-20260930 on1342calls; Inception provider.1136items18October+206items19October (sourceToronto dates). First-attempt measurements, failed deliveries remain failures. NOT a held-out scored release.37author-development cases permanently excluded from release diagnostics; dependent variants/correlated routes and unbalanced31families. Six new domains have only8questions each. No independent-trial inference or broad reliability claim. Mixed original dates/providers/request loads, not synchronized reruns. Median/p95 excludes retry/queue waits. Captured charges/prices are dated workload observations, not token tariffs. NOT a held-out scored release.37author-development cases permanently excluded from release diagnostics; dependent variants/correlated routes and unbalanced31families. Six new domains have only8questions each. No independent-trial inference or broad reliability claim. Mixed original dates/providers/request loads, not synchronized reruns. Median/p95 excludes retry/queue waits. Captured charges/prices are dated workload observations, not token tariffs. Checked / reported: 2026-10-20 · Samples: 67 Snapshot: 2026-10-19 · Revision: 419978f2753b4e3bb70a8be9aca7fe0c9e59bc9f · Published summary observations; upstream task/result terms retained Result source · Benchmark source
Decision reference DEVELOPMENT1342 · F09 · Exact truth counting	52.113%	Setup & source Decision reference DEVELOPMENT1342 · F09 · Exact truth counting Source consolidated development overview19October2026; Mercury Decide direct native Decision route through OpenRouter, returned inception/mercury-decide-20260930 on1342calls; Inception provider.1136items18October+206items19October (sourceToronto dates). First-attempt measurements, failed deliveries remain failures. NOT a held-out scored release.37author-development cases permanently excluded from release diagnostics; dependent variants/correlated routes and unbalanced31families. Six new domains have only8questions each. No independent-trial inference or broad reliability claim. Mixed original dates/providers/request loads, not synchronized reruns. Median/p95 excludes retry/queue waits. Captured charges/prices are dated workload observations, not token tariffs. NOT a held-out scored release.37author-development cases permanently excluded from release diagnostics; dependent variants/correlated routes and unbalanced31families. Six new domains have only8questions each. No independent-trial inference or broad reliability claim. Mixed original dates/providers/request loads, not synchronized reruns. Median/p95 excludes retry/queue waits. Captured charges/prices are dated workload observations, not token tariffs. Checked / reported: 2026-10-20 · Samples: 71 Snapshot: 2026-10-19 · Revision: 419978f2753b4e3bb70a8be9aca7fe0c9e59bc9f · Published summary observations; upstream task/result terms retained Result source · Benchmark source
Decision reference DEVELOPMENT1342 · F10 · Truth-teller consistency	52.703%	Setup & source Decision reference DEVELOPMENT1342 · F10 · Truth-teller consistency Source consolidated development overview19October2026; Mercury Decide direct native Decision route through OpenRouter, returned inception/mercury-decide-20260930 on1342calls; Inception provider.1136items18October+206items19October (sourceToronto dates). First-attempt measurements, failed deliveries remain failures. NOT a held-out scored release.37author-development cases permanently excluded from release diagnostics; dependent variants/correlated routes and unbalanced31families. Six new domains have only8questions each. No independent-trial inference or broad reliability claim. Mixed original dates/providers/request loads, not synchronized reruns. Median/p95 excludes retry/queue waits. Captured charges/prices are dated workload observations, not token tariffs. NOT a held-out scored release.37author-development cases permanently excluded from release diagnostics; dependent variants/correlated routes and unbalanced31families. Six new domains have only8questions each. No independent-trial inference or broad reliability claim. Mixed original dates/providers/request loads, not synchronized reruns. Median/p95 excludes retry/queue waits. Captured charges/prices are dated workload observations, not token tariffs. Checked / reported: 2026-10-20 · Samples: 74 Snapshot: 2026-10-19 · Revision: 419978f2753b4e3bb70a8be9aca7fe0c9e59bc9f · Published summary observations; upstream task/result terms retained Result source · Benchmark source
Decision reference DEVELOPMENT1342 · F11 · Sufficiency and inconsistency	73.171%	Setup & source Decision reference DEVELOPMENT1342 · F11 · Sufficiency and inconsistency Source consolidated development overview19October2026; Mercury Decide direct native Decision route through OpenRouter, returned inception/mercury-decide-20260930 on1342calls; Inception provider.1136items18October+206items19October (sourceToronto dates). First-attempt measurements, failed deliveries remain failures. NOT a held-out scored release.37author-development cases permanently excluded from release diagnostics; dependent variants/correlated routes and unbalanced31families. Six new domains have only8questions each. No independent-trial inference or broad reliability claim. Mixed original dates/providers/request loads, not synchronized reruns. Median/p95 excludes retry/queue waits. Captured charges/prices are dated workload observations, not token tariffs. NOT a held-out scored release.37author-development cases permanently excluded from release diagnostics; dependent variants/correlated routes and unbalanced31families. Six new domains have only8questions each. No independent-trial inference or broad reliability claim. Mixed original dates/providers/request loads, not synchronized reruns. Median/p95 excludes retry/queue waits. Captured charges/prices are dated workload observations, not token tariffs. Checked / reported: 2026-10-20 · Samples: 41 Snapshot: 2026-10-19 · Revision: 419978f2753b4e3bb70a8be9aca7fe0c9e59bc9f · Published summary observations; upstream task/result terms retained Result source · Benchmark source
Decision reference DEVELOPMENT1342 · F12 · Object swaps	65.385%	Setup & source Decision reference DEVELOPMENT1342 · F12 · Object swaps Source consolidated development overview19October2026; Mercury Decide direct native Decision route through OpenRouter, returned inception/mercury-decide-20260930 on1342calls; Inception provider.1136items18October+206items19October (sourceToronto dates). First-attempt measurements, failed deliveries remain failures. NOT a held-out scored release.37author-development cases permanently excluded from release diagnostics; dependent variants/correlated routes and unbalanced31families. Six new domains have only8questions each. No independent-trial inference or broad reliability claim. Mixed original dates/providers/request loads, not synchronized reruns. Median/p95 excludes retry/queue waits. Captured charges/prices are dated workload observations, not token tariffs. NOT a held-out scored release.37author-development cases permanently excluded from release diagnostics; dependent variants/correlated routes and unbalanced31families. Six new domains have only8questions each. No independent-trial inference or broad reliability claim. Mixed original dates/providers/request loads, not synchronized reruns. Median/p95 excludes retry/queue waits. Captured charges/prices are dated workload observations, not token tariffs. Checked / reported: 2026-10-20 · Samples: 52 Snapshot: 2026-10-19 · Revision: 419978f2753b4e3bb70a8be9aca7fe0c9e59bc9f · Published summary observations; upstream task/result terms retained Result source · Benchmark source
Decision reference DEVELOPMENT1342 · F13 · Sequential procedures	25.000%	Setup & source Decision reference DEVELOPMENT1342 · F13 · Sequential procedures Source consolidated development overview19October2026; Mercury Decide direct native Decision route through OpenRouter, returned inception/mercury-decide-20260930 on1342calls; Inception provider.1136items18October+206items19October (sourceToronto dates). First-attempt measurements, failed deliveries remain failures. NOT a held-out scored release.37author-development cases permanently excluded from release diagnostics; dependent variants/correlated routes and unbalanced31families. Six new domains have only8questions each. No independent-trial inference or broad reliability claim. Mixed original dates/providers/request loads, not synchronized reruns. Median/p95 excludes retry/queue waits. Captured charges/prices are dated workload observations, not token tariffs. NOT a held-out scored release.37author-development cases permanently excluded from release diagnostics; dependent variants/correlated routes and unbalanced31families. Six new domains have only8questions each. No independent-trial inference or broad reliability claim. Mixed original dates/providers/request loads, not synchronized reruns. Median/p95 excludes retry/queue waits. Captured charges/prices are dated workload observations, not token tariffs. Checked / reported: 2026-10-20 · Samples: 48 Snapshot: 2026-10-19 · Revision: 419978f2753b4e3bb70a8be9aca7fe0c9e59bc9f · Published summary observations; upstream task/result terms retained Result source · Benchmark source
Decision reference DEVELOPMENT1342 · F14 · Temporal intervals	83.333%	Setup & source Decision reference DEVELOPMENT1342 · F14 · Temporal intervals Source consolidated development overview19October2026; Mercury Decide direct native Decision route through OpenRouter, returned inception/mercury-decide-20260930 on1342calls; Inception provider.1136items18October+206items19October (sourceToronto dates). First-attempt measurements, failed deliveries remain failures. NOT a held-out scored release.37author-development cases permanently excluded from release diagnostics; dependent variants/correlated routes and unbalanced31families. Six new domains have only8questions each. No independent-trial inference or broad reliability claim. Mixed original dates/providers/request loads, not synchronized reruns. Median/p95 excludes retry/queue waits. Captured charges/prices are dated workload observations, not token tariffs. NOT a held-out scored release.37author-development cases permanently excluded from release diagnostics; dependent variants/correlated routes and unbalanced31families. Six new domains have only8questions each. No independent-trial inference or broad reliability claim. Mixed original dates/providers/request loads, not synchronized reruns. Median/p95 excludes retry/queue waits. Captured charges/prices are dated workload observations, not token tariffs. Checked / reported: 2026-10-20 · Samples: 42 Snapshot: 2026-10-19 · Revision: 419978f2753b4e3bb70a8be9aca7fe0c9e59bc9f · Published summary observations; upstream task/result terms retained Result source · Benchmark source
Decision reference DEVELOPMENT1342 · F15 · Spatial tracking	63.265%	Setup & source Decision reference DEVELOPMENT1342 · F15 · Spatial tracking Source consolidated development overview19October2026; Mercury Decide direct native Decision route through OpenRouter, returned inception/mercury-decide-20260930 on1342calls; Inception provider.1136items18October+206items19October (sourceToronto dates). First-attempt measurements, failed deliveries remain failures. NOT a held-out scored release.37author-development cases permanently excluded from release diagnostics; dependent variants/correlated routes and unbalanced31families. Six new domains have only8questions each. No independent-trial inference or broad reliability claim. Mixed original dates/providers/request loads, not synchronized reruns. Median/p95 excludes retry/queue waits. Captured charges/prices are dated workload observations, not token tariffs. NOT a held-out scored release.37author-development cases permanently excluded from release diagnostics; dependent variants/correlated routes and unbalanced31families. Six new domains have only8questions each. No independent-trial inference or broad reliability claim. Mixed original dates/providers/request loads, not synchronized reruns. Median/p95 excludes retry/queue waits. Captured charges/prices are dated workload observations, not token tariffs. Checked / reported: 2026-10-20 · Samples: 49 Snapshot: 2026-10-19 · Revision: 419978f2753b4e3bb70a8be9aca7fe0c9e59bc9f · Published summary observations; upstream task/result terms retained Result source · Benchmark source
Decision reference DEVELOPMENT1342 · F16 · Reference binding	70.000%	Setup & source Decision reference DEVELOPMENT1342 · F16 · Reference binding Source consolidated development overview19October2026; Mercury Decide direct native Decision route through OpenRouter, returned inception/mercury-decide-20260930 on1342calls; Inception provider.1136items18October+206items19October (sourceToronto dates). First-attempt measurements, failed deliveries remain failures. NOT a held-out scored release.37author-development cases permanently excluded from release diagnostics; dependent variants/correlated routes and unbalanced31families. Six new domains have only8questions each. No independent-trial inference or broad reliability claim. Mixed original dates/providers/request loads, not synchronized reruns. Median/p95 excludes retry/queue waits. Captured charges/prices are dated workload observations, not token tariffs. NOT a held-out scored release.37author-development cases permanently excluded from release diagnostics; dependent variants/correlated routes and unbalanced31families. Six new domains have only8questions each. No independent-trial inference or broad reliability claim. Mixed original dates/providers/request loads, not synchronized reruns. Median/p95 excludes retry/queue waits. Captured charges/prices are dated workload observations, not token tariffs. Checked / reported: 2026-10-20 · Samples: 50 Snapshot: 2026-10-19 · Revision: 419978f2753b4e3bb70a8be9aca7fe0c9e59bc9f · Published summary observations; upstream task/result terms retained Result source · Benchmark source
Decision reference DEVELOPMENT1342 · F17 · Beliefs and perspectives	86.000%	Setup & source Decision reference DEVELOPMENT1342 · F17 · Beliefs and perspectives Source consolidated development overview19October2026; Mercury Decide direct native Decision route through OpenRouter, returned inception/mercury-decide-20260930 on1342calls; Inception provider.1136items18October+206items19October (sourceToronto dates). First-attempt measurements, failed deliveries remain failures. NOT a held-out scored release.37author-development cases permanently excluded from release diagnostics; dependent variants/correlated routes and unbalanced31families. Six new domains have only8questions each. No independent-trial inference or broad reliability claim. Mixed original dates/providers/request loads, not synchronized reruns. Median/p95 excludes retry/queue waits. Captured charges/prices are dated workload observations, not token tariffs. NOT a held-out scored release.37author-development cases permanently excluded from release diagnostics; dependent variants/correlated routes and unbalanced31families. Six new domains have only8questions each. No independent-trial inference or broad reliability claim. Mixed original dates/providers/request loads, not synchronized reruns. Median/p95 excludes retry/queue waits. Captured charges/prices are dated workload observations, not token tariffs. Checked / reported: 2026-10-20 · Samples: 50 Snapshot: 2026-10-19 · Revision: 419978f2753b4e3bb70a8be9aca7fe0c9e59bc9f · Published summary observations; upstream task/result terms retained Result source · Benchmark source
Decision reference DEVELOPMENT1342 · F18 · Causal intervention	70.000%	Setup & source Decision reference DEVELOPMENT1342 · F18 · Causal intervention Source consolidated development overview19October2026; Mercury Decide direct native Decision route through OpenRouter, returned inception/mercury-decide-20260930 on1342calls; Inception provider.1136items18October+206items19October (sourceToronto dates). First-attempt measurements, failed deliveries remain failures. NOT a held-out scored release.37author-development cases permanently excluded from release diagnostics; dependent variants/correlated routes and unbalanced31families. Six new domains have only8questions each. No independent-trial inference or broad reliability claim. Mixed original dates/providers/request loads, not synchronized reruns. Median/p95 excludes retry/queue waits. Captured charges/prices are dated workload observations, not token tariffs. NOT a held-out scored release.37author-development cases permanently excluded from release diagnostics; dependent variants/correlated routes and unbalanced31families. Six new domains have only8questions each. No independent-trial inference or broad reliability claim. Mixed original dates/providers/request loads, not synchronized reruns. Median/p95 excludes retry/queue waits. Captured charges/prices are dated workload observations, not token tariffs. Checked / reported: 2026-10-20 · Samples: 50 Snapshot: 2026-10-19 · Revision: 419978f2753b4e3bb70a8be9aca7fe0c9e59bc9f · Published summary observations; upstream task/result terms retained Result source · Benchmark source
Decision reference DEVELOPMENT1342 · F19 · Exceptions and priorities	94.828%	Setup & source Decision reference DEVELOPMENT1342 · F19 · Exceptions and priorities Source consolidated development overview19October2026; Mercury Decide direct native Decision route through OpenRouter, returned inception/mercury-decide-20260930 on1342calls; Inception provider.1136items18October+206items19October (sourceToronto dates). First-attempt measurements, failed deliveries remain failures. NOT a held-out scored release.37author-development cases permanently excluded from release diagnostics; dependent variants/correlated routes and unbalanced31families. Six new domains have only8questions each. No independent-trial inference or broad reliability claim. Mixed original dates/providers/request loads, not synchronized reruns. Median/p95 excludes retry/queue waits. Captured charges/prices are dated workload observations, not token tariffs. NOT a held-out scored release.37author-development cases permanently excluded from release diagnostics; dependent variants/correlated routes and unbalanced31families. Six new domains have only8questions each. No independent-trial inference or broad reliability claim. Mixed original dates/providers/request loads, not synchronized reruns. Median/p95 excludes retry/queue waits. Captured charges/prices are dated workload observations, not token tariffs. Checked / reported: 2026-10-20 · Samples: 58 Snapshot: 2026-10-19 · Revision: 419978f2753b4e3bb70a8be9aca7fe0c9e59bc9f · Published summary observations; upstream task/result terms retained Result source · Benchmark source
Decision reference DEVELOPMENT1342 · F20 · Planning	88.372%	Setup & source Decision reference DEVELOPMENT1342 · F20 · Planning Source consolidated development overview19October2026; Mercury Decide direct native Decision route through OpenRouter, returned inception/mercury-decide-20260930 on1342calls; Inception provider.1136items18October+206items19October (sourceToronto dates). First-attempt measurements, failed deliveries remain failures. NOT a held-out scored release.37author-development cases permanently excluded from release diagnostics; dependent variants/correlated routes and unbalanced31families. Six new domains have only8questions each. No independent-trial inference or broad reliability claim. Mixed original dates/providers/request loads, not synchronized reruns. Median/p95 excludes retry/queue waits. Captured charges/prices are dated workload observations, not token tariffs. NOT a held-out scored release.37author-development cases permanently excluded from release diagnostics; dependent variants/correlated routes and unbalanced31families. Six new domains have only8questions each. No independent-trial inference or broad reliability claim. Mixed original dates/providers/request loads, not synchronized reruns. Median/p95 excludes retry/queue waits. Captured charges/prices are dated workload observations, not token tariffs. Checked / reported: 2026-10-20 · Samples: 43 Snapshot: 2026-10-19 · Revision: 419978f2753b4e3bb70a8be9aca7fe0c9e59bc9f · Published summary observations; upstream task/result terms retained Result source · Benchmark source
Decision reference DEVELOPMENT1342 · F21 · Directed reachability	93.023%	Setup & source Decision reference DEVELOPMENT1342 · F21 · Directed reachability Source consolidated development overview19October2026; Mercury Decide direct native Decision route through OpenRouter, returned inception/mercury-decide-20260930 on1342calls; Inception provider.1136items18October+206items19October (sourceToronto dates). First-attempt measurements, failed deliveries remain failures. NOT a held-out scored release.37author-development cases permanently excluded from release diagnostics; dependent variants/correlated routes and unbalanced31families. Six new domains have only8questions each. No independent-trial inference or broad reliability claim. Mixed original dates/providers/request loads, not synchronized reruns. Median/p95 excludes retry/queue waits. Captured charges/prices are dated workload observations, not token tariffs. NOT a held-out scored release.37author-development cases permanently excluded from release diagnostics; dependent variants/correlated routes and unbalanced31families. Six new domains have only8questions each. No independent-trial inference or broad reliability claim. Mixed original dates/providers/request loads, not synchronized reruns. Median/p95 excludes retry/queue waits. Captured charges/prices are dated workload observations, not token tariffs. Checked / reported: 2026-10-20 · Samples: 43 Snapshot: 2026-10-19 · Revision: 419978f2753b4e3bb70a8be9aca7fe0c9e59bc9f · Published summary observations; upstream task/result terms retained Result source · Benchmark source
Decision reference DEVELOPMENT1342 · F22 · Preference ordering	92.157%	Setup & source Decision reference DEVELOPMENT1342 · F22 · Preference ordering Source consolidated development overview19October2026; Mercury Decide direct native Decision route through OpenRouter, returned inception/mercury-decide-20260930 on1342calls; Inception provider.1136items18October+206items19October (sourceToronto dates). First-attempt measurements, failed deliveries remain failures. NOT a held-out scored release.37author-development cases permanently excluded from release diagnostics; dependent variants/correlated routes and unbalanced31families. Six new domains have only8questions each. No independent-trial inference or broad reliability claim. Mixed original dates/providers/request loads, not synchronized reruns. Median/p95 excludes retry/queue waits. Captured charges/prices are dated workload observations, not token tariffs. NOT a held-out scored release.37author-development cases permanently excluded from release diagnostics; dependent variants/correlated routes and unbalanced31families. Six new domains have only8questions each. No independent-trial inference or broad reliability claim. Mixed original dates/providers/request loads, not synchronized reruns. Median/p95 excludes retry/queue waits. Captured charges/prices are dated workload observations, not token tariffs. Checked / reported: 2026-10-20 · Samples: 51 Snapshot: 2026-10-19 · Revision: 419978f2753b4e3bb70a8be9aca7fe0c9e59bc9f · Published summary observations; upstream task/result terms retained Result source · Benchmark source
Decision reference DEVELOPMENT1342 · F23 · Rule counterexamples	98.077%	Setup & source Decision reference DEVELOPMENT1342 · F23 · Rule counterexamples Source consolidated development overview19October2026; Mercury Decide direct native Decision route through OpenRouter, returned inception/mercury-decide-20260930 on1342calls; Inception provider.1136items18October+206items19October (sourceToronto dates). First-attempt measurements, failed deliveries remain failures. NOT a held-out scored release.37author-development cases permanently excluded from release diagnostics; dependent variants/correlated routes and unbalanced31families. Six new domains have only8questions each. No independent-trial inference or broad reliability claim. Mixed original dates/providers/request loads, not synchronized reruns. Median/p95 excludes retry/queue waits. Captured charges/prices are dated workload observations, not token tariffs. NOT a held-out scored release.37author-development cases permanently excluded from release diagnostics; dependent variants/correlated routes and unbalanced31families. Six new domains have only8questions each. No independent-trial inference or broad reliability claim. Mixed original dates/providers/request loads, not synchronized reruns. Median/p95 excludes retry/queue waits. Captured charges/prices are dated workload observations, not token tariffs. Checked / reported: 2026-10-20 · Samples: 52 Snapshot: 2026-10-19 · Revision: 419978f2753b4e3bb70a8be9aca7fe0c9e59bc9f · Published summary observations; upstream task/result terms retained Result source · Benchmark source
Decision reference DEVELOPMENT1342 · F24 · Authority and stale records	93.023%	Setup & source Decision reference DEVELOPMENT1342 · F24 · Authority and stale records Source consolidated development overview19October2026; Mercury Decide direct native Decision route through OpenRouter, returned inception/mercury-decide-20260930 on1342calls; Inception provider.1136items18October+206items19October (sourceToronto dates). First-attempt measurements, failed deliveries remain failures. NOT a held-out scored release.37author-development cases permanently excluded from release diagnostics; dependent variants/correlated routes and unbalanced31families. Six new domains have only8questions each. No independent-trial inference or broad reliability claim. Mixed original dates/providers/request loads, not synchronized reruns. Median/p95 excludes retry/queue waits. Captured charges/prices are dated workload observations, not token tariffs. NOT a held-out scored release.37author-development cases permanently excluded from release diagnostics; dependent variants/correlated routes and unbalanced31families. Six new domains have only8questions each. No independent-trial inference or broad reliability claim. Mixed original dates/providers/request loads, not synchronized reruns. Median/p95 excludes retry/queue waits. Captured charges/prices are dated workload observations, not token tariffs. Checked / reported: 2026-10-20 · Samples: 43 Snapshot: 2026-10-19 · Revision: 419978f2753b4e3bb70a8be9aca7fe0c9e59bc9f · Published summary observations; upstream task/result terms retained Result source · Benchmark source
Decision reference DEVELOPMENT1342 · F25 · Fault diagnosis	92.000%	Setup & source Decision reference DEVELOPMENT1342 · F25 · Fault diagnosis Source consolidated development overview19October2026; Mercury Decide direct native Decision route through OpenRouter, returned inception/mercury-decide-20260930 on1342calls; Inception provider.1136items18October+206items19October (sourceToronto dates). First-attempt measurements, failed deliveries remain failures. NOT a held-out scored release.37author-development cases permanently excluded from release diagnostics; dependent variants/correlated routes and unbalanced31families. Six new domains have only8questions each. No independent-trial inference or broad reliability claim. Mixed original dates/providers/request loads, not synchronized reruns. Median/p95 excludes retry/queue waits. Captured charges/prices are dated workload observations, not token tariffs. NOT a held-out scored release.37author-development cases permanently excluded from release diagnostics; dependent variants/correlated routes and unbalanced31families. Six new domains have only8questions each. No independent-trial inference or broad reliability claim. Mixed original dates/providers/request loads, not synchronized reruns. Median/p95 excludes retry/queue waits. Captured charges/prices are dated workload observations, not token tariffs. Checked / reported: 2026-10-20 · Samples: 50 Snapshot: 2026-10-19 · Revision: 419978f2753b4e3bb70a8be9aca7fe0c9e59bc9f · Published summary observations; upstream task/result terms retained Result source · Benchmark source
Decision reference DEVELOPMENT1342 · F26 · Probabilistic risk	62.500%	Setup & source Decision reference DEVELOPMENT1342 · F26 · Probabilistic risk Source consolidated development overview19October2026; Mercury Decide direct native Decision route through OpenRouter, returned inception/mercury-decide-20260930 on1342calls; Inception provider.1136items18October+206items19October (sourceToronto dates). First-attempt measurements, failed deliveries remain failures. NOT a held-out scored release.37author-development cases permanently excluded from release diagnostics; dependent variants/correlated routes and unbalanced31families. Six new domains have only8questions each. No independent-trial inference or broad reliability claim. Mixed original dates/providers/request loads, not synchronized reruns. Median/p95 excludes retry/queue waits. Captured charges/prices are dated workload observations, not token tariffs. NOT a held-out scored release.37author-development cases permanently excluded from release diagnostics; dependent variants/correlated routes and unbalanced31families. Six new domains have only8questions each. No independent-trial inference or broad reliability claim. Mixed original dates/providers/request loads, not synchronized reruns. Median/p95 excludes retry/queue waits. Captured charges/prices are dated workload observations, not token tariffs. Checked / reported: 2026-10-20 · Samples: 8 Snapshot: 2026-10-19 · Revision: 419978f2753b4e3bb70a8be9aca7fe0c9e59bc9f · Published summary observations; upstream task/result terms retained Result source · Benchmark source
Decision reference DEVELOPMENT1342 · F27 · Value of information	37.500%	Setup & source Decision reference DEVELOPMENT1342 · F27 · Value of information Source consolidated development overview19October2026; Mercury Decide direct native Decision route through OpenRouter, returned inception/mercury-decide-20260930 on1342calls; Inception provider.1136items18October+206items19October (sourceToronto dates). First-attempt measurements, failed deliveries remain failures. NOT a held-out scored release.37author-development cases permanently excluded from release diagnostics; dependent variants/correlated routes and unbalanced31families. Six new domains have only8questions each. No independent-trial inference or broad reliability claim. Mixed original dates/providers/request loads, not synchronized reruns. Median/p95 excludes retry/queue waits. Captured charges/prices are dated workload observations, not token tariffs. NOT a held-out scored release.37author-development cases permanently excluded from release diagnostics; dependent variants/correlated routes and unbalanced31families. Six new domains have only8questions each. No independent-trial inference or broad reliability claim. Mixed original dates/providers/request loads, not synchronized reruns. Median/p95 excludes retry/queue waits. Captured charges/prices are dated workload observations, not token tariffs. Checked / reported: 2026-10-20 · Samples: 8 Snapshot: 2026-10-19 · Revision: 419978f2753b4e3bb70a8be9aca7fe0c9e59bc9f · Published summary observations; upstream task/result terms retained Result source · Benchmark source
Decision reference DEVELOPMENT1342 · F28 · Robust decisions without a prior	37.500%	Setup & source Decision reference DEVELOPMENT1342 · F28 · Robust decisions without a prior Source consolidated development overview19October2026; Mercury Decide direct native Decision route through OpenRouter, returned inception/mercury-decide-20260930 on1342calls; Inception provider.1136items18October+206items19October (sourceToronto dates). First-attempt measurements, failed deliveries remain failures. NOT a held-out scored release.37author-development cases permanently excluded from release diagnostics; dependent variants/correlated routes and unbalanced31families. Six new domains have only8questions each. No independent-trial inference or broad reliability claim. Mixed original dates/providers/request loads, not synchronized reruns. Median/p95 excludes retry/queue waits. Captured charges/prices are dated workload observations, not token tariffs. NOT a held-out scored release.37author-development cases permanently excluded from release diagnostics; dependent variants/correlated routes and unbalanced31families. Six new domains have only8questions each. No independent-trial inference or broad reliability claim. Mixed original dates/providers/request loads, not synchronized reruns. Median/p95 excludes retry/queue waits. Captured charges/prices are dated workload observations, not token tariffs. Checked / reported: 2026-10-20 · Samples: 8 Snapshot: 2026-10-19 · Revision: 419978f2753b4e3bb70a8be9aca7fe0c9e59bc9f · Published summary observations; upstream task/result terms retained Result source · Benchmark source
Decision reference DEVELOPMENT1342 · F29 · Strategic decisions	62.500%	Setup & source Decision reference DEVELOPMENT1342 · F29 · Strategic decisions Source consolidated development overview19October2026; Mercury Decide direct native Decision route through OpenRouter, returned inception/mercury-decide-20260930 on1342calls; Inception provider.1136items18October+206items19October (sourceToronto dates). First-attempt measurements, failed deliveries remain failures. NOT a held-out scored release.37author-development cases permanently excluded from release diagnostics; dependent variants/correlated routes and unbalanced31families. Six new domains have only8questions each. No independent-trial inference or broad reliability claim. Mixed original dates/providers/request loads, not synchronized reruns. Median/p95 excludes retry/queue waits. Captured charges/prices are dated workload observations, not token tariffs. NOT a held-out scored release.37author-development cases permanently excluded from release diagnostics; dependent variants/correlated routes and unbalanced31families. Six new domains have only8questions each. No independent-trial inference or broad reliability claim. Mixed original dates/providers/request loads, not synchronized reruns. Median/p95 excludes retry/queue waits. Captured charges/prices are dated workload observations, not token tariffs. Checked / reported: 2026-10-20 · Samples: 8 Snapshot: 2026-10-19 · Revision: 419978f2753b4e3bb70a8be9aca7fe0c9e59bc9f · Published summary observations; upstream task/result terms retained Result source · Benchmark source
Decision reference DEVELOPMENT1342 · F30 · Concurrent-history consistency	62.500%	Setup & source Decision reference DEVELOPMENT1342 · F30 · Concurrent-history consistency Source consolidated development overview19October2026; Mercury Decide direct native Decision route through OpenRouter, returned inception/mercury-decide-20260930 on1342calls; Inception provider.1136items18October+206items19October (sourceToronto dates). First-attempt measurements, failed deliveries remain failures. NOT a held-out scored release.37author-development cases permanently excluded from release diagnostics; dependent variants/correlated routes and unbalanced31families. Six new domains have only8questions each. No independent-trial inference or broad reliability claim. Mixed original dates/providers/request loads, not synchronized reruns. Median/p95 excludes retry/queue waits. Captured charges/prices are dated workload observations, not token tariffs. NOT a held-out scored release.37author-development cases permanently excluded from release diagnostics; dependent variants/correlated routes and unbalanced31families. Six new domains have only8questions each. No independent-trial inference or broad reliability claim. Mixed original dates/providers/request loads, not synchronized reruns. Median/p95 excludes retry/queue waits. Captured charges/prices are dated workload observations, not token tariffs. Checked / reported: 2026-10-20 · Samples: 8 Snapshot: 2026-10-19 · Revision: 419978f2753b4e3bb70a8be9aca7fe0c9e59bc9f · Published summary observations; upstream task/result terms retained Result source · Benchmark source
Decision reference DEVELOPMENT1342 · F31 · Information-flow security	62.500%	Setup & source Decision reference DEVELOPMENT1342 · F31 · Information-flow security Source consolidated development overview19October2026; Mercury Decide direct native Decision route through OpenRouter, returned inception/mercury-decide-20260930 on1342calls; Inception provider.1136items18October+206items19October (sourceToronto dates). First-attempt measurements, failed deliveries remain failures. NOT a held-out scored release.37author-development cases permanently excluded from release diagnostics; dependent variants/correlated routes and unbalanced31families. Six new domains have only8questions each. No independent-trial inference or broad reliability claim. Mixed original dates/providers/request loads, not synchronized reruns. Median/p95 excludes retry/queue waits. Captured charges/prices are dated workload observations, not token tariffs. NOT a held-out scored release.37author-development cases permanently excluded from release diagnostics; dependent variants/correlated routes and unbalanced31families. Six new domains have only8questions each. No independent-trial inference or broad reliability claim. Mixed original dates/providers/request loads, not synchronized reruns. Median/p95 excludes retry/queue waits. Captured charges/prices are dated workload observations, not token tariffs. Checked / reported: 2026-10-20 · Samples: 8 Snapshot: 2026-10-19 · Revision: 419978f2753b4e3bb70a8be9aca7fe0c9e59bc9f · Published summary observations; upstream task/result terms retained Result source · Benchmark source
Decision reference DEVELOPMENT1342 · release-exclusion-diagnostic	77.241%	Setup & source Decision reference DEVELOPMENT1342 · release-exclusion-diagnostic Source consolidated development overview19October2026; Mercury Decide direct native Decision route through OpenRouter, returned inception/mercury-decide-20260930 on1342calls; Inception provider.1136items18October+206items19October (sourceToronto dates). First-attempt measurements, failed deliveries remain failures. NOT a held-out scored release.37author-development cases permanently excluded from release diagnostics; dependent variants/correlated routes and unbalanced31families. Six new domains have only8questions each. No independent-trial inference or broad reliability claim. Mixed original dates/providers/request loads, not synchronized reruns. Median/p95 excludes retry/queue waits. Captured charges/prices are dated workload observations, not token tariffs. NOT a held-out scored release.37author-development cases permanently excluded from release diagnostics; dependent variants/correlated routes and unbalanced31families. Six new domains have only8questions each. No independent-trial inference or broad reliability claim. Mixed original dates/providers/request loads, not synchronized reruns. Median/p95 excludes retry/queue waits. Captured charges/prices are dated workload observations, not token tariffs. Checked / reported: 2026-10-20 · Samples: 1305 Snapshot: 2026-10-19 · Revision: 419978f2753b4e3bb70a8be9aca7fe0c9e59bc9f · Published summary observations; upstream task/result terms retained Result source · Benchmark source
Decision reference DEVELOPMENT1342 · new-domains	54.167%	Setup & source Decision reference DEVELOPMENT1342 · new-domains Source consolidated development overview19October2026; Mercury Decide direct native Decision route through OpenRouter, returned inception/mercury-decide-20260930 on1342calls; Inception provider.1136items18October+206items19October (sourceToronto dates). First-attempt measurements, failed deliveries remain failures. NOT a held-out scored release.37author-development cases permanently excluded from release diagnostics; dependent variants/correlated routes and unbalanced31families. Six new domains have only8questions each. No independent-trial inference or broad reliability claim. Mixed original dates/providers/request loads, not synchronized reruns. Median/p95 excludes retry/queue waits. Captured charges/prices are dated workload observations, not token tariffs. NOT a held-out scored release.37author-development cases permanently excluded from release diagnostics; dependent variants/correlated routes and unbalanced31families. Six new domains have only8questions each. No independent-trial inference or broad reliability claim. Mixed original dates/providers/request loads, not synchronized reruns. Median/p95 excludes retry/queue waits. Captured charges/prices are dated workload observations, not token tariffs. Checked / reported: 2026-10-20 · Samples: 48 Snapshot: 2026-10-19 · Revision: 419978f2753b4e3bb70a8be9aca7fe0c9e59bc9f · Published summary observations; upstream task/result terms retained Result source · Benchmark source
Decision reference DEVELOPMENT1342 · Confidence≥0.8 · selected accuracy	89.815%	Setup & source Decision reference DEVELOPMENT1342 · Confidence≥0.8 · selected accuracy Source consolidated development overview19October2026; Mercury Decide direct native Decision route through OpenRouter, returned inception/mercury-decide-20260930 on1342calls; Inception provider.1136items18October+206items19October (sourceToronto dates). First-attempt measurements, failed deliveries remain failures. NOT a held-out scored release.37author-development cases permanently excluded from release diagnostics; dependent variants/correlated routes and unbalanced31families. Six new domains have only8questions each. No independent-trial inference or broad reliability claim. Mixed original dates/providers/request loads, not synchronized reruns. Median/p95 excludes retry/queue waits. Captured charges/prices are dated workload observations, not token tariffs. NOT a held-out scored release.37author-development cases permanently excluded from release diagnostics; dependent variants/correlated routes and unbalanced31families. Six new domains have only8questions each. No independent-trial inference or broad reliability claim. Mixed original dates/providers/request loads, not synchronized reruns. Median/p95 excludes retry/queue waits. Captured charges/prices are dated workload observations, not token tariffs. Checked / reported: 2026-10-20 · Samples: 972 Snapshot: 2026-10-19 · Revision: 419978f2753b4e3bb70a8be9aca7fe0c9e59bc9f · Published summary observations; upstream task/result terms retained Result source · Benchmark source