Examenos

Exact named hosted API observation; All-case accuracy.

Named source route upstage/solar-decide; ModelTested runner v1; tested_on=openrouter. Backend weight revision and API inference precision unpublished.

Publisher generated 2026-11-01; board added this route 2026-10-11. Execution dates, token usage and hashes remain in its per-run receipts. No private prompt/response archive imported.

56 authored situations, including 19 hard cases, each with two typed questions; 14 public and 42 hidden cases. Source aggregate averages per-case decision accuracy. Confidence score is 100×(1−mean multiclass Brier/2), not ECE or a calibration probability. Small single-run benchmark; current HF access/size does not inherit this evaluator’s stale closed/API-only metadata. No tariff inferred from rounded cost 0.0. Rows added 1 November 2026 use revision 668a52ad3b932ca910fd985f96dcdf0a85d2dae1, where the earlier 11 systems are unchanged. Rows added 2 November 2026 use revision 5c6c19bb3b7332ccd28c8c13bd32cdc69cedc8f0.

Checked / reported: 2026-11-01 · Samples: 56

Snapshot: 2026-10-18 · Revision: 081e8f37130a4a9077b8cffe62827d4c3b9e3b23; 668a52ad3b932ca910fd985f96dcdf0a85d2dae1 for Drex Max, Jev 1.14 and d1.1; 5c6c19bb3b7332ccd28c8c13bd32cdc69cedc8f0 for Mammoth, Elephant, Joey, Span-02 and both Solar Decide routes · Published aggregate observations; result reuse terms not separately specified

Exact named hosted API observation; Hard-case accuracy.

Named source route upstage/solar-decide; ModelTested runner v1; tested_on=openrouter. Backend weight revision and API inference precision unpublished.

Publisher generated 2026-11-01; board added this route 2026-10-11. Execution dates, token usage and hashes remain in its per-run receipts. No private prompt/response archive imported.

56 authored situations, including 19 hard cases, each with two typed questions; 14 public and 42 hidden cases. Source aggregate averages per-case decision accuracy. Confidence score is 100×(1−mean multiclass Brier/2), not ECE or a calibration probability. Small single-run benchmark; current HF access/size does not inherit this evaluator’s stale closed/API-only metadata. No tariff inferred from rounded cost 0.0. Rows added 1 November 2026 use revision 668a52ad3b932ca910fd985f96dcdf0a85d2dae1, where the earlier 11 systems are unchanged. Rows added 2 November 2026 use revision 5c6c19bb3b7332ccd28c8c13bd32cdc69cedc8f0.

Checked / reported: 2026-11-01 · Samples: 56

Snapshot: 2026-10-18 · Revision: 081e8f37130a4a9077b8cffe62827d4c3b9e3b23; 668a52ad3b932ca910fd985f96dcdf0a85d2dae1 for Drex Max, Jev 1.14 and d1.1; 5c6c19bb3b7332ccd28c8c13bd32cdc69cedc8f0 for Mammoth, Elephant, Joey, Span-02 and both Solar Decide routes · Published aggregate observations; result reuse terms not separately specified

Exact named hosted API observation; Public-case accuracy.

Named source route upstage/solar-decide; ModelTested runner v1; tested_on=openrouter. Backend weight revision and API inference precision unpublished.

Publisher generated 2026-11-01; board added this route 2026-10-11. Execution dates, token usage and hashes remain in its per-run receipts. No private prompt/response archive imported.

56 authored situations, including 19 hard cases, each with two typed questions; 14 public and 42 hidden cases. Source aggregate averages per-case decision accuracy. Confidence score is 100×(1−mean multiclass Brier/2), not ECE or a calibration probability. Small single-run benchmark; current HF access/size does not inherit this evaluator’s stale closed/API-only metadata. No tariff inferred from rounded cost 0.0. Rows added 1 November 2026 use revision 668a52ad3b932ca910fd985f96dcdf0a85d2dae1, where the earlier 11 systems are unchanged. Rows added 2 November 2026 use revision 5c6c19bb3b7332ccd28c8c13bd32cdc69cedc8f0.

Checked / reported: 2026-11-01 · Samples: 56

Snapshot: 2026-10-18 · Revision: 081e8f37130a4a9077b8cffe62827d4c3b9e3b23; 668a52ad3b932ca910fd985f96dcdf0a85d2dae1 for Drex Max, Jev 1.14 and d1.1; 5c6c19bb3b7332ccd28c8c13bd32cdc69cedc8f0 for Mammoth, Elephant, Joey, Span-02 and both Solar Decide routes · Published aggregate observations; result reuse terms not separately specified

Exact named hosted API observation; Hidden-case accuracy.

Named source route upstage/solar-decide; ModelTested runner v1; tested_on=openrouter. Backend weight revision and API inference precision unpublished.

Publisher generated 2026-11-01; board added this route 2026-10-11. Execution dates, token usage and hashes remain in its per-run receipts. No private prompt/response archive imported.

56 authored situations, including 19 hard cases, each with two typed questions; 14 public and 42 hidden cases. Source aggregate averages per-case decision accuracy. Confidence score is 100×(1−mean multiclass Brier/2), not ECE or a calibration probability. Small single-run benchmark; current HF access/size does not inherit this evaluator’s stale closed/API-only metadata. No tariff inferred from rounded cost 0.0. Rows added 1 November 2026 use revision 668a52ad3b932ca910fd985f96dcdf0a85d2dae1, where the earlier 11 systems are unchanged. Rows added 2 November 2026 use revision 5c6c19bb3b7332ccd28c8c13bd32cdc69cedc8f0.

Checked / reported: 2026-11-01 · Samples: 56

Snapshot: 2026-10-18 · Revision: 081e8f37130a4a9077b8cffe62827d4c3b9e3b23; 668a52ad3b932ca910fd985f96dcdf0a85d2dae1 for Drex Max, Jev 1.14 and d1.1; 5c6c19bb3b7332ccd28c8c13bd32cdc69cedc8f0 for Mammoth, Elephant, Joey, Span-02 and both Solar Decide routes · Published aggregate observations; result reuse terms not separately specified

Exact named hosted API observation; Brier-based confidence score.

Named source route upstage/solar-decide; ModelTested runner v1; tested_on=openrouter. Backend weight revision and API inference precision unpublished.

Publisher generated 2026-11-01; board added this route 2026-10-11. Execution dates, token usage and hashes remain in its per-run receipts. No private prompt/response archive imported.

56 authored situations, including 19 hard cases, each with two typed questions; 14 public and 42 hidden cases. Source aggregate averages per-case decision accuracy. Confidence score is 100×(1−mean multiclass Brier/2), not ECE or a calibration probability. Small single-run benchmark; current HF access/size does not inherit this evaluator’s stale closed/API-only metadata. No tariff inferred from rounded cost 0.0. Rows added 1 November 2026 use revision 668a52ad3b932ca910fd985f96dcdf0a85d2dae1, where the earlier 11 systems are unchanged. Rows added 2 November 2026 use revision 5c6c19bb3b7332ccd28c8c13bd32cdc69cedc8f0.

Checked / reported: 2026-11-01 · Samples: 56

Snapshot: 2026-10-18 · Revision: 081e8f37130a4a9077b8cffe62827d4c3b9e3b23; 668a52ad3b932ca910fd985f96dcdf0a85d2dae1 for Drex Max, Jev 1.14 and d1.1; 5c6c19bb3b7332ccd28c8c13bd32cdc69cedc8f0 for Mammoth, Elephant, Joey, Span-02 and both Solar Decide routes · Published aggregate observations; result reuse terms not separately specified

JevBench Headline: aegis2 · accuracy

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "aegis2", "p50 ms": "832", "p95 ms": "14916", "p99 ms": "19386", "mean input tokens": "505", "total input tokens": "252,268", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:38-0700 · Samples: 500 · Source 95% CI: 77.4–84.2

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Headline: aegis2_response · accuracy

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "aegis2_response", "p50 ms": "633", "p95 ms": "15815", "p99 ms": "17801", "mean input tokens": "607", "total input tokens": "303,429", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:38-0700 · Samples: 500 · Source 95% CI: 71.8–79.2

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Headline: atbench500 · accuracy

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "atbench500", "p50 ms": "7869", "p95 ms": "16261", "p99 ms": "18906", "mean input tokens": "2223", "total input tokens": "1,111,555", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:38-0700 · Samples: 500 · Source 95% CI: 69.8–77.60000000000001

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Headline: helpsteer2_helpfulness · mae_expected_level

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "helpsteer2_helpfulness", "p50 ms": "11283", "p95 ms": "19955", "p99 ms": "22113", "mean input tokens": "934", "total input tokens": "467,105", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:38-0700 · Samples: 500 · Source 95% CI: 0.987–1.16

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Headline: jailbreak_classification · accuracy

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "jailbreak_classification", "p50 ms": "638", "p95 ms": "5906", "p99 ms": "16716", "mean input tokens": "713", "total input tokens": "285,150", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:38-0700 · Samples: 400 · Source 95% CI: 94.5–98.3

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Headline: mcpbench_tool_selection · tool_selection_accuracy

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "mcpbench_tool_selection", "p50 ms": "939", "p95 ms": "6360", "p99 ms": "8673", "mean input tokens": "915", "total input tokens": "23,778", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:38-0700 · Samples: 26

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Headline: mcpmark_server_routing · server_routing_accuracy

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "mcpmark_server_routing", "p50 ms": "10413", "p95 ms": "17797", "p99 ms": "18316", "mean input tokens": "470", "total input tokens": "159,016", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:38-0700 · Samples: 338 · Source 95% CI: 41.099999999999994–55.300000000000004

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Headline: mcpmark_tool_selection · tool_selection_accuracy

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "mcpmark_tool_selection", "p50 ms": "582", "p95 ms": "4044", "p99 ms": "4044", "mean input tokens": "647", "total input tokens": "11,642", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:38-0700 · Samples: 18

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Headline: medmcqa · accuracy

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "medmcqa", "p50 ms": "789", "p95 ms": "17032", "p99 ms": "20599", "mean input tokens": "399", "total input tokens": "399,370", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:38-0700 · Samples: 1000 · Source 95% CI: 68.60000000000001–75.5

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Headline: medqa_usmle · accuracy

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "medqa_usmle", "p50 ms": "646", "p95 ms": "15886", "p99 ms": "17428", "mean input tokens": "554", "total input tokens": "553,650", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:38-0700 · Samples: 1000 · Source 95% CI: 80.2–86

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Headline: mmlu_pro · accuracy

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "mmlu_pro", "p50 ms": "4439", "p95 ms": "16695", "p99 ms": "18851", "mean input tokens": "545", "total input tokens": "544,710", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:38-0700 · Samples: 1000 · Source 95% CI: 68.89999999999999–76.1

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Headline: nfcorpus_rank_choice · ndcg@10

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "nfcorpus_rank_choice", "p50 ms": "10892", "p95 ms": "18233", "p99 ms": "20540", "mean input tokens": "2503", "total input tokens": "720,835", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:38-0700 · Samples: 298 · Source 95% CI: 27.200000000000003–33.800000000000004

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Headline: nfcorpus_rank_graded · ndcg@10

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "nfcorpus_rank_graded", "p50 ms": "2075", "p95 ms": "13342", "p99 ms": "16494", "mean input tokens": "570", "total input tokens": "2,807,330", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:38-0700 · Samples: 4924 · Source 95% CI: 32.1–39.1

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Headline: nfcorpus_rank_noul · ndcg@10

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "nfcorpus_rank_noul", "p50 ms": "12751", "p95 ms": "25054", "p99 ms": "30135", "mean input tokens": "48457", "total input tokens": "14,440,038", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:38-0700 · Samples: 298 · Source 95% CI: 28.9–35.9

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Headline: prompt_injections · accuracy

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "prompt_injections", "p50 ms": "1601", "p95 ms": "15486", "p99 ms": "20707", "mean input tokens": "497", "total input tokens": "57,662", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:38-0700 · Samples: 116 · Source 95% CI: 62.1–78.4

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Headline: pubmedqa · accuracy

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "pubmedqa", "p50 ms": "895", "p95 ms": "16201", "p99 ms": "19088", "mean input tokens": "711", "total input tokens": "355,398", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:38-0700 · Samples: 500 · Source 95% CI: 55.00000000000001–63.6

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Headline: rewardbench2 · accuracy

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "rewardbench2", "p50 ms": "12406", "p95 ms": "16064", "p99 ms": "17608", "mean input tokens": "1957", "total input tokens": "2,348,908", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:38-0700 · Samples: 1200 · Source 95% CI: 71.89999999999999–77.8

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Headline: scienceqa_text · accuracy

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "scienceqa_text", "p50 ms": "605", "p95 ms": "7884", "p99 ms": "15687", "mean input tokens": "402", "total input tokens": "402,396", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:38-0700 · Samples: 1000 · Source 95% CI: 94.8–97.5

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Headline: scifact_rank_choice · ndcg@10

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "scifact_rank_choice", "p50 ms": "1427", "p95 ms": "18610", "p99 ms": "23746", "mean input tokens": "2897", "total input tokens": "869,148", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:38-0700 · Samples: 300 · Source 95% CI: 58.3–67.80000000000001

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Headline: scifact_rank_noul · ndcg@10

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "scifact_rank_noul", "p50 ms": "13122", "p95 ms": "22947", "p99 ms": "26821", "mean input tokens": "61557", "total input tokens": "18,467,080", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:38-0700 · Samples: 300 · Source 95% CI: 50.4–61

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Headline: sst5 · mae_expected_level

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "sst5", "p50 ms": "650", "p95 ms": "13828", "p99 ms": "15148", "mean input tokens": "390", "total input tokens": "194,814", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:38-0700 · Samples: 500 · Source 95% CI: 0.538–0.66

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: mcpbench_tool_selection · accuracy

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "mcpbench_tool_selection", "p50 ms": "939", "p95 ms": "6360", "p99 ms": "8673", "mean input tokens": "915", "total input tokens": "23,778", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:38-0700 · Samples: 26

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: mcpbench_tool_selection · macro-F1

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "mcpbench_tool_selection", "p50 ms": "939", "p95 ms": "6360", "p99 ms": "8673", "mean input tokens": "915", "total input tokens": "23,778", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:38-0700 · Samples: 26

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: mcpbench_tool_selection · Brier

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "mcpbench_tool_selection", "p50 ms": "939", "p95 ms": "6360", "p99 ms": "8673", "mean input tokens": "915", "total input tokens": "23,778", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:38-0700 · Samples: 26

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: mcpbench_tool_selection · NLL

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "mcpbench_tool_selection", "p50 ms": "939", "p95 ms": "6360", "p99 ms": "8673", "mean input tokens": "915", "total input tokens": "23,778", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:38-0700 · Samples: 26

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: mcpbench_tool_selection · ECE (10 bins)

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "mcpbench_tool_selection", "p50 ms": "939", "p95 ms": "6360", "p99 ms": "8673", "mean input tokens": "915", "total input tokens": "23,778", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:38-0700 · Samples: 26

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: mcpmark_server_routing · accuracy

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "mcpmark_server_routing", "p50 ms": "10413", "p95 ms": "17797", "p99 ms": "18316", "mean input tokens": "470", "total input tokens": "159,016", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:38-0700 · Samples: 338

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: mcpmark_server_routing · macro-F1

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "mcpmark_server_routing", "p50 ms": "10413", "p95 ms": "17797", "p99 ms": "18316", "mean input tokens": "470", "total input tokens": "159,016", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:38-0700 · Samples: 338

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: mcpmark_server_routing · Brier

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "mcpmark_server_routing", "p50 ms": "10413", "p95 ms": "17797", "p99 ms": "18316", "mean input tokens": "470", "total input tokens": "159,016", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:38-0700 · Samples: 338

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: mcpmark_server_routing · NLL

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "mcpmark_server_routing", "p50 ms": "10413", "p95 ms": "17797", "p99 ms": "18316", "mean input tokens": "470", "total input tokens": "159,016", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:38-0700 · Samples: 338

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: mcpmark_server_routing · ECE (10 bins)

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "mcpmark_server_routing", "p50 ms": "10413", "p95 ms": "17797", "p99 ms": "18316", "mean input tokens": "470", "total input tokens": "159,016", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:38-0700 · Samples: 338

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: mcpmark_tool_selection · accuracy

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "mcpmark_tool_selection", "p50 ms": "582", "p95 ms": "4044", "p99 ms": "4044", "mean input tokens": "647", "total input tokens": "11,642", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:38-0700 · Samples: 18

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: mcpmark_tool_selection · macro-F1

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "mcpmark_tool_selection", "p50 ms": "582", "p95 ms": "4044", "p99 ms": "4044", "mean input tokens": "647", "total input tokens": "11,642", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:38-0700 · Samples: 18

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: mcpmark_tool_selection · Brier

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "mcpmark_tool_selection", "p50 ms": "582", "p95 ms": "4044", "p99 ms": "4044", "mean input tokens": "647", "total input tokens": "11,642", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:38-0700 · Samples: 18

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: mcpmark_tool_selection · NLL

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "mcpmark_tool_selection", "p50 ms": "582", "p95 ms": "4044", "p99 ms": "4044", "mean input tokens": "647", "total input tokens": "11,642", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:38-0700 · Samples: 18

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: mcpmark_tool_selection · ECE (10 bins)

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "mcpmark_tool_selection", "p50 ms": "582", "p95 ms": "4044", "p99 ms": "4044", "mean input tokens": "647", "total input tokens": "11,642", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:38-0700 · Samples: 18

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: medmcqa · accuracy

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "medmcqa", "p50 ms": "789", "p95 ms": "17032", "p99 ms": "20599", "mean input tokens": "399", "total input tokens": "399,370", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:38-0700 · Samples: 1000

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: medmcqa · macro-F1

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "medmcqa", "p50 ms": "789", "p95 ms": "17032", "p99 ms": "20599", "mean input tokens": "399", "total input tokens": "399,370", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:38-0700 · Samples: 1000

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: medmcqa · Brier

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "medmcqa", "p50 ms": "789", "p95 ms": "17032", "p99 ms": "20599", "mean input tokens": "399", "total input tokens": "399,370", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:38-0700 · Samples: 1000

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: medmcqa · NLL

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "medmcqa", "p50 ms": "789", "p95 ms": "17032", "p99 ms": "20599", "mean input tokens": "399", "total input tokens": "399,370", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:38-0700 · Samples: 1000

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: medmcqa · ECE (10 bins)

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "medmcqa", "p50 ms": "789", "p95 ms": "17032", "p99 ms": "20599", "mean input tokens": "399", "total input tokens": "399,370", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:38-0700 · Samples: 1000

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: medqa_usmle · accuracy

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "medqa_usmle", "p50 ms": "646", "p95 ms": "15886", "p99 ms": "17428", "mean input tokens": "554", "total input tokens": "553,650", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:38-0700 · Samples: 1000

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: medqa_usmle · macro-F1

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "medqa_usmle", "p50 ms": "646", "p95 ms": "15886", "p99 ms": "17428", "mean input tokens": "554", "total input tokens": "553,650", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:38-0700 · Samples: 1000

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: medqa_usmle · Brier

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "medqa_usmle", "p50 ms": "646", "p95 ms": "15886", "p99 ms": "17428", "mean input tokens": "554", "total input tokens": "553,650", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:38-0700 · Samples: 1000

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: medqa_usmle · NLL

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "medqa_usmle", "p50 ms": "646", "p95 ms": "15886", "p99 ms": "17428", "mean input tokens": "554", "total input tokens": "553,650", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:38-0700 · Samples: 1000

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: medqa_usmle · ECE (10 bins)

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "medqa_usmle", "p50 ms": "646", "p95 ms": "15886", "p99 ms": "17428", "mean input tokens": "554", "total input tokens": "553,650", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:38-0700 · Samples: 1000

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: mmlu_pro · accuracy

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "mmlu_pro", "p50 ms": "4439", "p95 ms": "16695", "p99 ms": "18851", "mean input tokens": "545", "total input tokens": "544,710", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:38-0700 · Samples: 1000

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: mmlu_pro · macro-F1

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "mmlu_pro", "p50 ms": "4439", "p95 ms": "16695", "p99 ms": "18851", "mean input tokens": "545", "total input tokens": "544,710", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:38-0700 · Samples: 1000

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: mmlu_pro · Brier

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "mmlu_pro", "p50 ms": "4439", "p95 ms": "16695", "p99 ms": "18851", "mean input tokens": "545", "total input tokens": "544,710", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:38-0700 · Samples: 1000

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: mmlu_pro · NLL

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "mmlu_pro", "p50 ms": "4439", "p95 ms": "16695", "p99 ms": "18851", "mean input tokens": "545", "total input tokens": "544,710", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:38-0700 · Samples: 1000

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: mmlu_pro · ECE (10 bins)

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "mmlu_pro", "p50 ms": "4439", "p95 ms": "16695", "p99 ms": "18851", "mean input tokens": "545", "total input tokens": "544,710", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:38-0700 · Samples: 1000

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: pubmedqa · accuracy

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "pubmedqa", "p50 ms": "895", "p95 ms": "16201", "p99 ms": "19088", "mean input tokens": "711", "total input tokens": "355,398", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:38-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: pubmedqa · macro-F1

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "pubmedqa", "p50 ms": "895", "p95 ms": "16201", "p99 ms": "19088", "mean input tokens": "711", "total input tokens": "355,398", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:38-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: pubmedqa · Brier

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "pubmedqa", "p50 ms": "895", "p95 ms": "16201", "p99 ms": "19088", "mean input tokens": "711", "total input tokens": "355,398", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:38-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: pubmedqa · NLL

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "pubmedqa", "p50 ms": "895", "p95 ms": "16201", "p99 ms": "19088", "mean input tokens": "711", "total input tokens": "355,398", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:38-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: pubmedqa · ECE (10 bins)

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "pubmedqa", "p50 ms": "895", "p95 ms": "16201", "p99 ms": "19088", "mean input tokens": "711", "total input tokens": "355,398", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:38-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: rewardbench2 · accuracy

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "rewardbench2", "p50 ms": "12406", "p95 ms": "16064", "p99 ms": "17608", "mean input tokens": "1957", "total input tokens": "2,348,908", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:38-0700 · Samples: 1200

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: rewardbench2 · macro-F1

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "rewardbench2", "p50 ms": "12406", "p95 ms": "16064", "p99 ms": "17608", "mean input tokens": "1957", "total input tokens": "2,348,908", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:38-0700 · Samples: 1200

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: rewardbench2 · Brier

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "rewardbench2", "p50 ms": "12406", "p95 ms": "16064", "p99 ms": "17608", "mean input tokens": "1957", "total input tokens": "2,348,908", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:38-0700 · Samples: 1200

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: rewardbench2 · NLL

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "rewardbench2", "p50 ms": "12406", "p95 ms": "16064", "p99 ms": "17608", "mean input tokens": "1957", "total input tokens": "2,348,908", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:38-0700 · Samples: 1200

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: rewardbench2 · ECE (10 bins)

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "rewardbench2", "p50 ms": "12406", "p95 ms": "16064", "p99 ms": "17608", "mean input tokens": "1957", "total input tokens": "2,348,908", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:38-0700 · Samples: 1200

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: scienceqa_text · accuracy

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "scienceqa_text", "p50 ms": "605", "p95 ms": "7884", "p99 ms": "15687", "mean input tokens": "402", "total input tokens": "402,396", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:38-0700 · Samples: 1000

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: scienceqa_text · macro-F1

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "scienceqa_text", "p50 ms": "605", "p95 ms": "7884", "p99 ms": "15687", "mean input tokens": "402", "total input tokens": "402,396", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:38-0700 · Samples: 1000

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: scienceqa_text · Brier

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "scienceqa_text", "p50 ms": "605", "p95 ms": "7884", "p99 ms": "15687", "mean input tokens": "402", "total input tokens": "402,396", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:38-0700 · Samples: 1000

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: scienceqa_text · NLL

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "scienceqa_text", "p50 ms": "605", "p95 ms": "7884", "p99 ms": "15687", "mean input tokens": "402", "total input tokens": "402,396", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:38-0700 · Samples: 1000

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: scienceqa_text · ECE (10 bins)

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "scienceqa_text", "p50 ms": "605", "p95 ms": "7884", "p99 ms": "15687", "mean input tokens": "402", "total input tokens": "402,396", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:38-0700 · Samples: 1000

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: aegis2 · acc @0.5

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "aegis2", "p50 ms": "832", "p95 ms": "14916", "p99 ms": "19386", "mean input tokens": "505", "total input tokens": "252,268", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:38-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: aegis2 · acc if flipped

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "aegis2", "p50 ms": "832", "p95 ms": "14916", "p99 ms": "19386", "mean input tokens": "505", "total input tokens": "252,268", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:38-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: aegis2 · positive recall

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "aegis2", "p50 ms": "832", "p95 ms": "14916", "p99 ms": "19386", "mean input tokens": "505", "total input tokens": "252,268", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:38-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: aegis2 · FPR

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "aegis2", "p50 ms": "832", "p95 ms": "14916", "p99 ms": "19386", "mean input tokens": "505", "total input tokens": "252,268", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:38-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: aegis2 · AUROC

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "aegis2", "p50 ms": "832", "p95 ms": "14916", "p99 ms": "19386", "mean input tokens": "505", "total input tokens": "252,268", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:38-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: aegis2 · Brier

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "aegis2", "p50 ms": "832", "p95 ms": "14916", "p99 ms": "19386", "mean input tokens": "505", "total input tokens": "252,268", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:38-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: aegis2 · ECE

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "aegis2", "p50 ms": "832", "p95 ms": "14916", "p99 ms": "19386", "mean input tokens": "505", "total input tokens": "252,268", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:38-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: aegis2_response · acc @0.5

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "aegis2_response", "p50 ms": "633", "p95 ms": "15815", "p99 ms": "17801", "mean input tokens": "607", "total input tokens": "303,429", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:38-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: aegis2_response · acc if flipped

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "aegis2_response", "p50 ms": "633", "p95 ms": "15815", "p99 ms": "17801", "mean input tokens": "607", "total input tokens": "303,429", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:38-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: aegis2_response · positive recall

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "aegis2_response", "p50 ms": "633", "p95 ms": "15815", "p99 ms": "17801", "mean input tokens": "607", "total input tokens": "303,429", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:38-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: aegis2_response · FPR

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "aegis2_response", "p50 ms": "633", "p95 ms": "15815", "p99 ms": "17801", "mean input tokens": "607", "total input tokens": "303,429", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:38-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: aegis2_response · AUROC

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "aegis2_response", "p50 ms": "633", "p95 ms": "15815", "p99 ms": "17801", "mean input tokens": "607", "total input tokens": "303,429", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:38-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: aegis2_response · Brier

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "aegis2_response", "p50 ms": "633", "p95 ms": "15815", "p99 ms": "17801", "mean input tokens": "607", "total input tokens": "303,429", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:38-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: aegis2_response · ECE

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "aegis2_response", "p50 ms": "633", "p95 ms": "15815", "p99 ms": "17801", "mean input tokens": "607", "total input tokens": "303,429", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:38-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: atbench500 · acc @0.5

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "atbench500", "p50 ms": "7869", "p95 ms": "16261", "p99 ms": "18906", "mean input tokens": "2223", "total input tokens": "1,111,555", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:38-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: atbench500 · acc if flipped

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "atbench500", "p50 ms": "7869", "p95 ms": "16261", "p99 ms": "18906", "mean input tokens": "2223", "total input tokens": "1,111,555", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:38-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: atbench500 · positive recall

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "atbench500", "p50 ms": "7869", "p95 ms": "16261", "p99 ms": "18906", "mean input tokens": "2223", "total input tokens": "1,111,555", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:38-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: atbench500 · FPR

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "atbench500", "p50 ms": "7869", "p95 ms": "16261", "p99 ms": "18906", "mean input tokens": "2223", "total input tokens": "1,111,555", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:38-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: atbench500 · AUROC

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "atbench500", "p50 ms": "7869", "p95 ms": "16261", "p99 ms": "18906", "mean input tokens": "2223", "total input tokens": "1,111,555", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:38-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: atbench500 · Brier

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "atbench500", "p50 ms": "7869", "p95 ms": "16261", "p99 ms": "18906", "mean input tokens": "2223", "total input tokens": "1,111,555", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:38-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: atbench500 · ECE

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "atbench500", "p50 ms": "7869", "p95 ms": "16261", "p99 ms": "18906", "mean input tokens": "2223", "total input tokens": "1,111,555", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:38-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: jailbreak_classification · acc @0.5

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "jailbreak_classification", "p50 ms": "638", "p95 ms": "5906", "p99 ms": "16716", "mean input tokens": "713", "total input tokens": "285,150", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:38-0700 · Samples: 400

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: jailbreak_classification · acc if flipped

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "jailbreak_classification", "p50 ms": "638", "p95 ms": "5906", "p99 ms": "16716", "mean input tokens": "713", "total input tokens": "285,150", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:38-0700 · Samples: 400

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: jailbreak_classification · positive recall

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "jailbreak_classification", "p50 ms": "638", "p95 ms": "5906", "p99 ms": "16716", "mean input tokens": "713", "total input tokens": "285,150", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:38-0700 · Samples: 400

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: jailbreak_classification · FPR

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "jailbreak_classification", "p50 ms": "638", "p95 ms": "5906", "p99 ms": "16716", "mean input tokens": "713", "total input tokens": "285,150", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:38-0700 · Samples: 400

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: jailbreak_classification · AUROC

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "jailbreak_classification", "p50 ms": "638", "p95 ms": "5906", "p99 ms": "16716", "mean input tokens": "713", "total input tokens": "285,150", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:38-0700 · Samples: 400

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: jailbreak_classification · Brier

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "jailbreak_classification", "p50 ms": "638", "p95 ms": "5906", "p99 ms": "16716", "mean input tokens": "713", "total input tokens": "285,150", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:38-0700 · Samples: 400

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: jailbreak_classification · ECE

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "jailbreak_classification", "p50 ms": "638", "p95 ms": "5906", "p99 ms": "16716", "mean input tokens": "713", "total input tokens": "285,150", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:38-0700 · Samples: 400

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: prompt_injections · acc @0.5

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "prompt_injections", "p50 ms": "1601", "p95 ms": "15486", "p99 ms": "20707", "mean input tokens": "497", "total input tokens": "57,662", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:38-0700 · Samples: 116

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: prompt_injections · acc if flipped

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "prompt_injections", "p50 ms": "1601", "p95 ms": "15486", "p99 ms": "20707", "mean input tokens": "497", "total input tokens": "57,662", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:38-0700 · Samples: 116

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: prompt_injections · positive recall

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "prompt_injections", "p50 ms": "1601", "p95 ms": "15486", "p99 ms": "20707", "mean input tokens": "497", "total input tokens": "57,662", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:38-0700 · Samples: 116

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: prompt_injections · FPR

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "prompt_injections", "p50 ms": "1601", "p95 ms": "15486", "p99 ms": "20707", "mean input tokens": "497", "total input tokens": "57,662", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:38-0700 · Samples: 116

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: prompt_injections · AUROC

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "prompt_injections", "p50 ms": "1601", "p95 ms": "15486", "p99 ms": "20707", "mean input tokens": "497", "total input tokens": "57,662", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:38-0700 · Samples: 116

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: prompt_injections · Brier

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "prompt_injections", "p50 ms": "1601", "p95 ms": "15486", "p99 ms": "20707", "mean input tokens": "497", "total input tokens": "57,662", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:38-0700 · Samples: 116

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: prompt_injections · ECE

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "prompt_injections", "p50 ms": "1601", "p95 ms": "15486", "p99 ms": "20707", "mean input tokens": "497", "total input tokens": "57,662", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:38-0700 · Samples: 116

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ordinal slices: helpsteer2_helpfulness · exact (rounded)

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "helpsteer2_helpfulness", "p50 ms": "11283", "p95 ms": "19955", "p99 ms": "22113", "mean input tokens": "934", "total input tokens": "467,105", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:38-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ordinal slices: helpsteer2_helpfulness · within 1 level

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "helpsteer2_helpfulness", "p50 ms": "11283", "p95 ms": "19955", "p99 ms": "22113", "mean input tokens": "934", "total input tokens": "467,105", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:38-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ordinal slices: helpsteer2_helpfulness · MAE (expected level)

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "helpsteer2_helpfulness", "p50 ms": "11283", "p95 ms": "19955", "p99 ms": "22113", "mean input tokens": "934", "total input tokens": "467,105", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:38-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ordinal slices: helpsteer2_helpfulness · RPS

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "helpsteer2_helpfulness", "p50 ms": "11283", "p95 ms": "19955", "p99 ms": "22113", "mean input tokens": "934", "total input tokens": "467,105", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:38-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ordinal slices: helpsteer2_helpfulness · Spearman

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "helpsteer2_helpfulness", "p50 ms": "11283", "p95 ms": "19955", "p99 ms": "22113", "mean input tokens": "934", "total input tokens": "467,105", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:38-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ordinal slices: nfcorpus_rank_graded · exact (rounded)

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "nfcorpus_rank_graded", "p50 ms": "2075", "p95 ms": "13342", "p99 ms": "16494", "mean input tokens": "570", "total input tokens": "2,807,330", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:38-0700 · Samples: 4924

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ordinal slices: nfcorpus_rank_graded · within 1 level

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "nfcorpus_rank_graded", "p50 ms": "2075", "p95 ms": "13342", "p99 ms": "16494", "mean input tokens": "570", "total input tokens": "2,807,330", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:38-0700 · Samples: 4924

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ordinal slices: nfcorpus_rank_graded · MAE (expected level)

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "nfcorpus_rank_graded", "p50 ms": "2075", "p95 ms": "13342", "p99 ms": "16494", "mean input tokens": "570", "total input tokens": "2,807,330", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:38-0700 · Samples: 4924

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ordinal slices: nfcorpus_rank_graded · RPS

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "nfcorpus_rank_graded", "p50 ms": "2075", "p95 ms": "13342", "p99 ms": "16494", "mean input tokens": "570", "total input tokens": "2,807,330", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:38-0700 · Samples: 4924

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ordinal slices: nfcorpus_rank_graded · Spearman

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "nfcorpus_rank_graded", "p50 ms": "2075", "p95 ms": "13342", "p99 ms": "16494", "mean input tokens": "570", "total input tokens": "2,807,330", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:38-0700 · Samples: 4924

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ordinal slices: sst5 · exact (rounded)

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "sst5", "p50 ms": "650", "p95 ms": "13828", "p99 ms": "15148", "mean input tokens": "390", "total input tokens": "194,814", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:38-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ordinal slices: sst5 · within 1 level

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "sst5", "p50 ms": "650", "p95 ms": "13828", "p99 ms": "15148", "mean input tokens": "390", "total input tokens": "194,814", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:38-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ordinal slices: sst5 · MAE (expected level)

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "sst5", "p50 ms": "650", "p95 ms": "13828", "p99 ms": "15148", "mean input tokens": "390", "total input tokens": "194,814", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:38-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ordinal slices: sst5 · RPS

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "sst5", "p50 ms": "650", "p95 ms": "13828", "p99 ms": "15148", "mean input tokens": "390", "total input tokens": "194,814", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:38-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ordinal slices: sst5 · Spearman

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "sst5", "p50 ms": "650", "p95 ms": "13828", "p99 ms": "15148", "mean input tokens": "390", "total input tokens": "194,814", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:38-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ranking slices: nfcorpus_rank_choice · nDCG@10

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "nfcorpus_rank_choice", "p50 ms": "10892", "p95 ms": "18233", "p99 ms": "20540", "mean input tokens": "2503", "total input tokens": "720,835", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:38-0700 · Samples: 288

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ranking slices: nfcorpus_rank_choice · MRR@10

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "nfcorpus_rank_choice", "p50 ms": "10892", "p95 ms": "18233", "p99 ms": "20540", "mean input tokens": "2503", "total input tokens": "720,835", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:38-0700 · Samples: 288

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ranking slices: nfcorpus_rank_choice · Recall@10

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "nfcorpus_rank_choice", "p50 ms": "10892", "p95 ms": "18233", "p99 ms": "20540", "mean input tokens": "2503", "total input tokens": "720,835", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:38-0700 · Samples: 288

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ranking slices: nfcorpus_rank_graded · nDCG@10

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "nfcorpus_rank_graded", "p50 ms": "2075", "p95 ms": "13342", "p99 ms": "16494", "mean input tokens": "570", "total input tokens": "2,807,330", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:38-0700 · Samples: 298

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ranking slices: nfcorpus_rank_graded · MRR@10

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "nfcorpus_rank_graded", "p50 ms": "2075", "p95 ms": "13342", "p99 ms": "16494", "mean input tokens": "570", "total input tokens": "2,807,330", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:38-0700 · Samples: 298

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ranking slices: nfcorpus_rank_graded · Recall@10

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "nfcorpus_rank_graded", "p50 ms": "2075", "p95 ms": "13342", "p99 ms": "16494", "mean input tokens": "570", "total input tokens": "2,807,330", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:38-0700 · Samples: 298

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ranking slices: nfcorpus_rank_noul · nDCG@10

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "nfcorpus_rank_noul", "p50 ms": "12751", "p95 ms": "25054", "p99 ms": "30135", "mean input tokens": "48457", "total input tokens": "14,440,038", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:38-0700 · Samples: 298

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ranking slices: nfcorpus_rank_noul · MRR@10

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "nfcorpus_rank_noul", "p50 ms": "12751", "p95 ms": "25054", "p99 ms": "30135", "mean input tokens": "48457", "total input tokens": "14,440,038", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:38-0700 · Samples: 298

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ranking slices: nfcorpus_rank_noul · Recall@10

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "nfcorpus_rank_noul", "p50 ms": "12751", "p95 ms": "25054", "p99 ms": "30135", "mean input tokens": "48457", "total input tokens": "14,440,038", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:38-0700 · Samples: 298

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ranking slices: scifact_rank_choice · nDCG@10

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "scifact_rank_choice", "p50 ms": "1427", "p95 ms": "18610", "p99 ms": "23746", "mean input tokens": "2897", "total input tokens": "869,148", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:38-0700 · Samples: 300

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ranking slices: scifact_rank_choice · MRR@10

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "scifact_rank_choice", "p50 ms": "1427", "p95 ms": "18610", "p99 ms": "23746", "mean input tokens": "2897", "total input tokens": "869,148", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:38-0700 · Samples: 300

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ranking slices: scifact_rank_choice · Recall@10

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "scifact_rank_choice", "p50 ms": "1427", "p95 ms": "18610", "p99 ms": "23746", "mean input tokens": "2897", "total input tokens": "869,148", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:38-0700 · Samples: 300

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ranking slices: scifact_rank_noul · nDCG@10

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "scifact_rank_noul", "p50 ms": "13122", "p95 ms": "22947", "p99 ms": "26821", "mean input tokens": "61557", "total input tokens": "18,467,080", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:38-0700 · Samples: 300

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ranking slices: scifact_rank_noul · MRR@10

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "scifact_rank_noul", "p50 ms": "13122", "p95 ms": "22947", "p99 ms": "26821", "mean input tokens": "61557", "total input tokens": "18,467,080", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:38-0700 · Samples: 300

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ranking slices: scifact_rank_noul · Recall@10

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "scifact_rank_noul", "p50 ms": "13122", "p95 ms": "22947", "p99 ms": "26821", "mean input tokens": "61557", "total input tokens": "18,467,080", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:38-0700 · Samples: 300

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench By subset: rewardbench2 · accuracy

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "rewardbench2", "p50 ms": "12406", "p95 ms": "16064", "p99 ms": "17608", "mean input tokens": "1957", "total input tokens": "2,348,908", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:38-0700 · Samples: 240

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench By subset: rewardbench2 · accuracy

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "rewardbench2", "p50 ms": "12406", "p95 ms": "16064", "p99 ms": "17608", "mean input tokens": "1957", "total input tokens": "2,348,908", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:38-0700 · Samples: 240

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench By subset: rewardbench2 · accuracy

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "rewardbench2", "p50 ms": "12406", "p95 ms": "16064", "p99 ms": "17608", "mean input tokens": "1957", "total input tokens": "2,348,908", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:38-0700 · Samples: 240

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench By subset: rewardbench2 · accuracy

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "rewardbench2", "p50 ms": "12406", "p95 ms": "16064", "p99 ms": "17608", "mean input tokens": "1957", "total input tokens": "2,348,908", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:38-0700 · Samples: 240

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench By subset: rewardbench2 · accuracy

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "rewardbench2", "p50 ms": "12406", "p95 ms": "16064", "p99 ms": "17608", "mean input tokens": "1957", "total input tokens": "2,348,908", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:38-0700 · Samples: 240

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: mcpbench_tool_selection · original order

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "mcpbench_tool_selection", "p50 ms": "939", "p95 ms": "6360", "p99 ms": "8673", "mean input tokens": "915", "total input tokens": "23,778", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:38-0700 · Samples: 4

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: mcpbench_tool_selection · permuted order

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "mcpbench_tool_selection", "p50 ms": "939", "p95 ms": "6360", "p99 ms": "8673", "mean input tokens": "915", "total input tokens": "23,778", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:38-0700 · Samples: 4

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: mcpbench_tool_selection · both correct

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "mcpbench_tool_selection", "p50 ms": "939", "p95 ms": "6360", "p99 ms": "8673", "mean input tokens": "915", "total input tokens": "23,778", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:38-0700 · Samples: 4

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: mcpbench_tool_selection · flipped by reordering

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "mcpbench_tool_selection", "p50 ms": "939", "p95 ms": "6360", "p99 ms": "8673", "mean input tokens": "915", "total input tokens": "23,778", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:38-0700 · Samples: 4

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: mcpmark_server_routing · original order

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "mcpmark_server_routing", "p50 ms": "10413", "p95 ms": "17797", "p99 ms": "18316", "mean input tokens": "470", "total input tokens": "159,016", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:38-0700 · Samples: 169

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: mcpmark_server_routing · permuted order

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "mcpmark_server_routing", "p50 ms": "10413", "p95 ms": "17797", "p99 ms": "18316", "mean input tokens": "470", "total input tokens": "159,016", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:38-0700 · Samples: 169

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: mcpmark_server_routing · both correct

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "mcpmark_server_routing", "p50 ms": "10413", "p95 ms": "17797", "p99 ms": "18316", "mean input tokens": "470", "total input tokens": "159,016", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:38-0700 · Samples: 169

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: mcpmark_server_routing · flipped by reordering

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "mcpmark_server_routing", "p50 ms": "10413", "p95 ms": "17797", "p99 ms": "18316", "mean input tokens": "470", "total input tokens": "159,016", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:38-0700 · Samples: 169

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: mcpmark_tool_selection · original order

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "mcpmark_tool_selection", "p50 ms": "582", "p95 ms": "4044", "p99 ms": "4044", "mean input tokens": "647", "total input tokens": "11,642", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:38-0700 · Samples: 3

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: mcpmark_tool_selection · permuted order

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "mcpmark_tool_selection", "p50 ms": "582", "p95 ms": "4044", "p99 ms": "4044", "mean input tokens": "647", "total input tokens": "11,642", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:38-0700 · Samples: 3

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: mcpmark_tool_selection · both correct

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "mcpmark_tool_selection", "p50 ms": "582", "p95 ms": "4044", "p99 ms": "4044", "mean input tokens": "647", "total input tokens": "11,642", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:38-0700 · Samples: 3

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: mcpmark_tool_selection · flipped by reordering

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "mcpmark_tool_selection", "p50 ms": "582", "p95 ms": "4044", "p99 ms": "4044", "mean input tokens": "647", "total input tokens": "11,642", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:38-0700 · Samples: 3

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: medmcqa · original order

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "medmcqa", "p50 ms": "789", "p95 ms": "17032", "p99 ms": "20599", "mean input tokens": "399", "total input tokens": "399,370", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:38-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: medmcqa · permuted order

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "medmcqa", "p50 ms": "789", "p95 ms": "17032", "p99 ms": "20599", "mean input tokens": "399", "total input tokens": "399,370", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:38-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: medmcqa · both correct

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "medmcqa", "p50 ms": "789", "p95 ms": "17032", "p99 ms": "20599", "mean input tokens": "399", "total input tokens": "399,370", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:38-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: medmcqa · flipped by reordering

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "medmcqa", "p50 ms": "789", "p95 ms": "17032", "p99 ms": "20599", "mean input tokens": "399", "total input tokens": "399,370", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:38-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: medqa_usmle · original order

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "medqa_usmle", "p50 ms": "646", "p95 ms": "15886", "p99 ms": "17428", "mean input tokens": "554", "total input tokens": "553,650", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:38-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: medqa_usmle · permuted order

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "medqa_usmle", "p50 ms": "646", "p95 ms": "15886", "p99 ms": "17428", "mean input tokens": "554", "total input tokens": "553,650", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:38-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: medqa_usmle · both correct

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "medqa_usmle", "p50 ms": "646", "p95 ms": "15886", "p99 ms": "17428", "mean input tokens": "554", "total input tokens": "553,650", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:38-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: medqa_usmle · flipped by reordering

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "medqa_usmle", "p50 ms": "646", "p95 ms": "15886", "p99 ms": "17428", "mean input tokens": "554", "total input tokens": "553,650", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:38-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: mmlu_pro · original order

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "mmlu_pro", "p50 ms": "4439", "p95 ms": "16695", "p99 ms": "18851", "mean input tokens": "545", "total input tokens": "544,710", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:38-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: mmlu_pro · permuted order

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "mmlu_pro", "p50 ms": "4439", "p95 ms": "16695", "p99 ms": "18851", "mean input tokens": "545", "total input tokens": "544,710", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:38-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: mmlu_pro · both correct

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "mmlu_pro", "p50 ms": "4439", "p95 ms": "16695", "p99 ms": "18851", "mean input tokens": "545", "total input tokens": "544,710", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:38-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: mmlu_pro · flipped by reordering

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "mmlu_pro", "p50 ms": "4439", "p95 ms": "16695", "p99 ms": "18851", "mean input tokens": "545", "total input tokens": "544,710", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:38-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: rewardbench2 · original order

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "rewardbench2", "p50 ms": "12406", "p95 ms": "16064", "p99 ms": "17608", "mean input tokens": "1957", "total input tokens": "2,348,908", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:38-0700 · Samples: 594

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: rewardbench2 · permuted order

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "rewardbench2", "p50 ms": "12406", "p95 ms": "16064", "p99 ms": "17608", "mean input tokens": "1957", "total input tokens": "2,348,908", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:38-0700 · Samples: 594

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: rewardbench2 · both correct

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "rewardbench2", "p50 ms": "12406", "p95 ms": "16064", "p99 ms": "17608", "mean input tokens": "1957", "total input tokens": "2,348,908", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:38-0700 · Samples: 594

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: rewardbench2 · flipped by reordering

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "rewardbench2", "p50 ms": "12406", "p95 ms": "16064", "p99 ms": "17608", "mean input tokens": "1957", "total input tokens": "2,348,908", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:38-0700 · Samples: 594

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: scienceqa_text · original order

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "scienceqa_text", "p50 ms": "605", "p95 ms": "7884", "p99 ms": "15687", "mean input tokens": "402", "total input tokens": "402,396", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:38-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: scienceqa_text · permuted order

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "scienceqa_text", "p50 ms": "605", "p95 ms": "7884", "p99 ms": "15687", "mean input tokens": "402", "total input tokens": "402,396", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:38-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: scienceqa_text · both correct

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "scienceqa_text", "p50 ms": "605", "p95 ms": "7884", "p99 ms": "15687", "mean input tokens": "402", "total input tokens": "402,396", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:38-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: scienceqa_text · flipped by reordering

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "scienceqa_text", "p50 ms": "605", "p95 ms": "7884", "p99 ms": "15687", "mean input tokens": "402", "total input tokens": "402,396", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:38-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Sep26 · Index

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published category aggregate. Zeroed by the source: ["BANKING77", "MCP-Bench routing"].

Publisher normalized per-dataset scores, averaged within six categories and then categories. Complete6/6; not Benchmark Heaven JevBench or an Examenos recomputation.

Checked / reported: 2026-11-02 · Samples: Not published

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench Sep26 · Science

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published category aggregate. Zeroed by the source: ["BANKING77", "MCP-Bench routing"].

Publisher normalized per-dataset scores, averaged within six categories and then categories. Complete6/6; not Benchmark Heaven JevBench or an Examenos recomputation.

Checked / reported: 2026-11-02 · Samples: Not published

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench Sep26 · Tool calling

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published category aggregate. Zeroed by the source: ["BANKING77", "MCP-Bench routing"].

Publisher normalized per-dataset scores, averaged within six categories and then categories. Complete6/6; not Benchmark Heaven JevBench or an Examenos recomputation.

Checked / reported: 2026-11-02 · Samples: Not published

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench Sep26 · Business

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published category aggregate. Zeroed by the source: ["BANKING77", "MCP-Bench routing"].

Publisher normalized per-dataset scores, averaged within six categories and then categories. Complete6/6; not Benchmark Heaven JevBench or an Examenos recomputation.

Checked / reported: 2026-11-02 · Samples: Not published

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench Sep26 · Selection

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published category aggregate. Zeroed by the source: ["BANKING77", "MCP-Bench routing"].

Publisher normalized per-dataset scores, averaged within six categories and then categories. Complete6/6; not Benchmark Heaven JevBench or an Examenos recomputation.

Checked / reported: 2026-11-02 · Samples: Not published

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench Sep26 · Ranking

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published category aggregate. Zeroed by the source: ["BANKING77", "MCP-Bench routing"].

Publisher normalized per-dataset scores, averaged within six categories and then categories. Complete6/6; not Benchmark Heaven JevBench or an Examenos recomputation.

Checked / reported: 2026-11-02 · Samples: Not published

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench Sep26 · Judging

Scored 2026-10-15T23:39:38-0700. Model requested `or-upstage-solar-decide`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published category aggregate. Zeroed by the source: ["BANKING77", "MCP-Bench routing"].

Publisher normalized per-dataset scores, averaged within six categories and then categories. Complete6/6; not Benchmark Heaven JevBench or an Examenos recomputation.

Checked / reported: 2026-11-02 · Samples: Not published

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · English baseline · overall · acc

Solar Decide, same named hosted route; English baseline; slice overall. {}

Source observations: {"n": 2016, "acc": 0.7944, "auroc": 0.7962}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 2016

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · English baseline · overall · auroc

Solar Decide, same named hosted route; English baseline; slice overall. {}

Source observations: {"n": 2016, "acc": 0.7944, "auroc": 0.7962}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 2016

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · English baseline · aegis2 · acc

Solar Decide, same named hosted route; English baseline; slice aegis2. {}

Source observations: {"n": 500, "acc": 0.808, "auroc": 0.8447, "yes": 0.364, "gold_yes": 0.5}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 500

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · English baseline · aegis2 · auroc

Solar Decide, same named hosted route; English baseline; slice aegis2. {}

Source observations: {"n": 500, "acc": 0.808, "auroc": 0.8447, "yes": 0.364, "gold_yes": 0.5}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 500

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · English baseline · aegis2 · yes

Solar Decide, same named hosted route; English baseline; slice aegis2. {}

Source observations: {"n": 500, "acc": 0.808, "auroc": 0.8447, "yes": 0.364, "gold_yes": 0.5}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 500

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · English baseline · aegis2_response · acc

Solar Decide, same named hosted route; English baseline; slice aegis2_response. {}

Source observations: {"n": 500, "acc": 0.754, "auroc": 0.8007, "yes": 0.298, "gold_yes": 0.5}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 500

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · English baseline · aegis2_response · auroc

Solar Decide, same named hosted route; English baseline; slice aegis2_response. {}

Source observations: {"n": 500, "acc": 0.754, "auroc": 0.8007, "yes": 0.298, "gold_yes": 0.5}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 500

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · English baseline · aegis2_response · yes

Solar Decide, same named hosted route; English baseline; slice aegis2_response. {}

Source observations: {"n": 500, "acc": 0.754, "auroc": 0.8007, "yes": 0.298, "gold_yes": 0.5}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 500

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · English baseline · atbench500 · acc

Solar Decide, same named hosted route; English baseline; slice atbench500. {}

Source observations: {"n": 500, "acc": 0.738, "auroc": 0.8011, "yes": 0.554, "gold_yes": 0.5}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 500

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · English baseline · atbench500 · auroc

Solar Decide, same named hosted route; English baseline; slice atbench500. {}

Source observations: {"n": 500, "acc": 0.738, "auroc": 0.8011, "yes": 0.554, "gold_yes": 0.5}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 500

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · English baseline · atbench500 · yes

Solar Decide, same named hosted route; English baseline; slice atbench500. {}

Source observations: {"n": 500, "acc": 0.738, "auroc": 0.8011, "yes": 0.554, "gold_yes": 0.5}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 500

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · English baseline · jailbreak_classification · acc

Solar Decide, same named hosted route; English baseline; slice jailbreak_classification. {}

Source observations: {"n": 400, "acc": 0.965, "auroc": 0.9495, "yes": 0.3275, "gold_yes": 0.3525}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 400

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · English baseline · jailbreak_classification · auroc

Solar Decide, same named hosted route; English baseline; slice jailbreak_classification. {}

Source observations: {"n": 400, "acc": 0.965, "auroc": 0.9495, "yes": 0.3275, "gold_yes": 0.3525}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 400

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · English baseline · jailbreak_classification · yes

Solar Decide, same named hosted route; English baseline; slice jailbreak_classification. {}

Source observations: {"n": 400, "acc": 0.965, "auroc": 0.9495, "yes": 0.3275, "gold_yes": 0.3525}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 400

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · English baseline · prompt_injections · acc

Solar Decide, same named hosted route; English baseline; slice prompt_injections. {}

Source observations: {"n": 116, "acc": 0.7069, "auroc": 0.5851, "yes": 0.2586, "gold_yes": 0.5172}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 116

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · English baseline · prompt_injections · auroc

Solar Decide, same named hosted route; English baseline; slice prompt_injections. {}

Source observations: {"n": 116, "acc": 0.7069, "auroc": 0.5851, "yes": 0.2586, "gold_yes": 0.5172}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 116

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · English baseline · prompt_injections · yes

Solar Decide, same named hosted route; English baseline; slice prompt_injections. {}

Source observations: {"n": 116, "acc": 0.7069, "auroc": 0.5851, "yes": 0.2586, "gold_yes": 0.5172}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 116

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Chinese · overall · acc

Solar Decide, same named hosted route; Chinese; slice overall. {"answered": 1979, "cases": 1979, "complete": true}

Source observations: {"n": 1979, "acc": 0.7514, "en_acc": 0.7949, "delta": -0.0435, "ci": [-0.064, -0.0238], "significant": true, "auroc": 0.7629, "en_auroc": 0.7975}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 1979

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Chinese · overall · auroc

Solar Decide, same named hosted route; Chinese; slice overall. {"answered": 1979, "cases": 1979, "complete": true}

Source observations: {"n": 1979, "acc": 0.7514, "en_acc": 0.7949, "delta": -0.0435, "ci": [-0.064, -0.0238], "significant": true, "auroc": 0.7629, "en_auroc": 0.7975}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 1979

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Chinese · overall · en_acc

Solar Decide, same named hosted route; Chinese; slice overall. {"answered": 1979, "cases": 1979, "complete": true}

Source observations: {"n": 1979, "acc": 0.7514, "en_acc": 0.7949, "delta": -0.0435, "ci": [-0.064, -0.0238], "significant": true, "auroc": 0.7629, "en_auroc": 0.7975}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 1979

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Chinese · overall · en_auroc

Solar Decide, same named hosted route; Chinese; slice overall. {"answered": 1979, "cases": 1979, "complete": true}

Source observations: {"n": 1979, "acc": 0.7514, "en_acc": 0.7949, "delta": -0.0435, "ci": [-0.064, -0.0238], "significant": true, "auroc": 0.7629, "en_auroc": 0.7975}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 1979

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Chinese · overall · delta

Solar Decide, same named hosted route; Chinese; slice overall. {"answered": 1979, "cases": 1979, "complete": true}

Source observations: {"n": 1979, "acc": 0.7514, "en_acc": 0.7949, "delta": -0.0435, "ci": [-0.064, -0.0238], "significant": true, "auroc": 0.7629, "en_auroc": 0.7975}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 1979

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Chinese · aegis2 · acc

Solar Decide, same named hosted route; Chinese; slice aegis2. {"answered": 1979, "cases": 1979, "complete": true}

Source observations: {"n": 499, "acc": 0.8397, "en_acc": 0.8076, "auroc": 0.868, "en_auroc": 0.8443, "yes": 0.4289, "en_yes": 0.3647, "gold_yes": 0.501}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 499

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Chinese · aegis2 · auroc

Solar Decide, same named hosted route; Chinese; slice aegis2. {"answered": 1979, "cases": 1979, "complete": true}

Source observations: {"n": 499, "acc": 0.8397, "en_acc": 0.8076, "auroc": 0.868, "en_auroc": 0.8443, "yes": 0.4289, "en_yes": 0.3647, "gold_yes": 0.501}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 499

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Chinese · aegis2 · en_acc

Solar Decide, same named hosted route; Chinese; slice aegis2. {"answered": 1979, "cases": 1979, "complete": true}

Source observations: {"n": 499, "acc": 0.8397, "en_acc": 0.8076, "auroc": 0.868, "en_auroc": 0.8443, "yes": 0.4289, "en_yes": 0.3647, "gold_yes": 0.501}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 499

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Chinese · aegis2 · en_auroc

Solar Decide, same named hosted route; Chinese; slice aegis2. {"answered": 1979, "cases": 1979, "complete": true}

Source observations: {"n": 499, "acc": 0.8397, "en_acc": 0.8076, "auroc": 0.868, "en_auroc": 0.8443, "yes": 0.4289, "en_yes": 0.3647, "gold_yes": 0.501}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 499

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Chinese · aegis2 · yes

Solar Decide, same named hosted route; Chinese; slice aegis2. {"answered": 1979, "cases": 1979, "complete": true}

Source observations: {"n": 499, "acc": 0.8397, "en_acc": 0.8076, "auroc": 0.868, "en_auroc": 0.8443, "yes": 0.4289, "en_yes": 0.3647, "gold_yes": 0.501}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 499

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Chinese · aegis2 · en_yes

Solar Decide, same named hosted route; Chinese; slice aegis2. {"answered": 1979, "cases": 1979, "complete": true}

Source observations: {"n": 499, "acc": 0.8397, "en_acc": 0.8076, "auroc": 0.868, "en_auroc": 0.8443, "yes": 0.4289, "en_yes": 0.3647, "gold_yes": 0.501}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 499

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Chinese · aegis2_response · acc

Solar Decide, same named hosted route; Chinese; slice aegis2_response. {"answered": 1979, "cases": 1979, "complete": true}

Source observations: {"n": 500, "acc": 0.762, "en_acc": 0.754, "auroc": 0.8181, "en_auroc": 0.8007, "yes": 0.346, "en_yes": 0.298, "gold_yes": 0.5}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 500

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Chinese · aegis2_response · auroc

Solar Decide, same named hosted route; Chinese; slice aegis2_response. {"answered": 1979, "cases": 1979, "complete": true}

Source observations: {"n": 500, "acc": 0.762, "en_acc": 0.754, "auroc": 0.8181, "en_auroc": 0.8007, "yes": 0.346, "en_yes": 0.298, "gold_yes": 0.5}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 500

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Chinese · aegis2_response · en_acc

Solar Decide, same named hosted route; Chinese; slice aegis2_response. {"answered": 1979, "cases": 1979, "complete": true}

Source observations: {"n": 500, "acc": 0.762, "en_acc": 0.754, "auroc": 0.8181, "en_auroc": 0.8007, "yes": 0.346, "en_yes": 0.298, "gold_yes": 0.5}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 500

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Chinese · aegis2_response · en_auroc

Solar Decide, same named hosted route; Chinese; slice aegis2_response. {"answered": 1979, "cases": 1979, "complete": true}

Source observations: {"n": 500, "acc": 0.762, "en_acc": 0.754, "auroc": 0.8181, "en_auroc": 0.8007, "yes": 0.346, "en_yes": 0.298, "gold_yes": 0.5}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 500

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Chinese · aegis2_response · yes

Solar Decide, same named hosted route; Chinese; slice aegis2_response. {"answered": 1979, "cases": 1979, "complete": true}

Source observations: {"n": 500, "acc": 0.762, "en_acc": 0.754, "auroc": 0.8181, "en_auroc": 0.8007, "yes": 0.346, "en_yes": 0.298, "gold_yes": 0.5}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 500

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Chinese · aegis2_response · en_yes

Solar Decide, same named hosted route; Chinese; slice aegis2_response. {"answered": 1979, "cases": 1979, "complete": true}

Source observations: {"n": 500, "acc": 0.762, "en_acc": 0.754, "auroc": 0.8181, "en_auroc": 0.8007, "yes": 0.346, "en_yes": 0.298, "gold_yes": 0.5}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 500

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Chinese · atbench500 · acc

Solar Decide, same named hosted route; Chinese; slice atbench500. {"answered": 1979, "cases": 1979, "complete": true}

Source observations: {"n": 467, "acc": 0.6124, "en_acc": 0.7388, "auroc": 0.6723, "en_auroc": 0.8023, "yes": 0.773, "en_yes": 0.5439, "gold_yes": 0.4925}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 467

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Chinese · atbench500 · auroc

Solar Decide, same named hosted route; Chinese; slice atbench500. {"answered": 1979, "cases": 1979, "complete": true}

Source observations: {"n": 467, "acc": 0.6124, "en_acc": 0.7388, "auroc": 0.6723, "en_auroc": 0.8023, "yes": 0.773, "en_yes": 0.5439, "gold_yes": 0.4925}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 467

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Chinese · atbench500 · en_acc

Solar Decide, same named hosted route; Chinese; slice atbench500. {"answered": 1979, "cases": 1979, "complete": true}

Source observations: {"n": 467, "acc": 0.6124, "en_acc": 0.7388, "auroc": 0.6723, "en_auroc": 0.8023, "yes": 0.773, "en_yes": 0.5439, "gold_yes": 0.4925}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 467

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Chinese · atbench500 · en_auroc

Solar Decide, same named hosted route; Chinese; slice atbench500. {"answered": 1979, "cases": 1979, "complete": true}

Source observations: {"n": 467, "acc": 0.6124, "en_acc": 0.7388, "auroc": 0.6723, "en_auroc": 0.8023, "yes": 0.773, "en_yes": 0.5439, "gold_yes": 0.4925}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 467

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Chinese · atbench500 · yes

Solar Decide, same named hosted route; Chinese; slice atbench500. {"answered": 1979, "cases": 1979, "complete": true}

Source observations: {"n": 467, "acc": 0.6124, "en_acc": 0.7388, "auroc": 0.6723, "en_auroc": 0.8023, "yes": 0.773, "en_yes": 0.5439, "gold_yes": 0.4925}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 467

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Chinese · atbench500 · en_yes

Solar Decide, same named hosted route; Chinese; slice atbench500. {"answered": 1979, "cases": 1979, "complete": true}

Source observations: {"n": 467, "acc": 0.6124, "en_acc": 0.7388, "auroc": 0.6723, "en_auroc": 0.8023, "yes": 0.773, "en_yes": 0.5439, "gold_yes": 0.4925}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 467

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Chinese · jailbreak_classification · acc

Solar Decide, same named hosted route; Chinese; slice jailbreak_classification. {"answered": 1979, "cases": 1979, "complete": true}

Source observations: {"n": 397, "acc": 0.9395, "en_acc": 0.9673, "auroc": 0.9345, "en_auroc": 0.9552, "yes": 0.3577, "en_yes": 0.3249, "gold_yes": 0.3476}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 397

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Chinese · jailbreak_classification · auroc

Solar Decide, same named hosted route; Chinese; slice jailbreak_classification. {"answered": 1979, "cases": 1979, "complete": true}

Source observations: {"n": 397, "acc": 0.9395, "en_acc": 0.9673, "auroc": 0.9345, "en_auroc": 0.9552, "yes": 0.3577, "en_yes": 0.3249, "gold_yes": 0.3476}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 397

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Chinese · jailbreak_classification · en_acc

Solar Decide, same named hosted route; Chinese; slice jailbreak_classification. {"answered": 1979, "cases": 1979, "complete": true}

Source observations: {"n": 397, "acc": 0.9395, "en_acc": 0.9673, "auroc": 0.9345, "en_auroc": 0.9552, "yes": 0.3577, "en_yes": 0.3249, "gold_yes": 0.3476}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 397

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Chinese · jailbreak_classification · en_auroc

Solar Decide, same named hosted route; Chinese; slice jailbreak_classification. {"answered": 1979, "cases": 1979, "complete": true}

Source observations: {"n": 397, "acc": 0.9395, "en_acc": 0.9673, "auroc": 0.9345, "en_auroc": 0.9552, "yes": 0.3577, "en_yes": 0.3249, "gold_yes": 0.3476}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 397

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Chinese · jailbreak_classification · yes

Solar Decide, same named hosted route; Chinese; slice jailbreak_classification. {"answered": 1979, "cases": 1979, "complete": true}

Source observations: {"n": 397, "acc": 0.9395, "en_acc": 0.9673, "auroc": 0.9345, "en_auroc": 0.9552, "yes": 0.3577, "en_yes": 0.3249, "gold_yes": 0.3476}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 397

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Chinese · jailbreak_classification · en_yes

Solar Decide, same named hosted route; Chinese; slice jailbreak_classification. {"answered": 1979, "cases": 1979, "complete": true}

Source observations: {"n": 397, "acc": 0.9395, "en_acc": 0.9673, "auroc": 0.9345, "en_auroc": 0.9552, "yes": 0.3577, "en_yes": 0.3249, "gold_yes": 0.3476}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 397

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Chinese · prompt_injections · acc

Solar Decide, same named hosted route; Chinese; slice prompt_injections. {"answered": 1979, "cases": 1979, "complete": true}

Source observations: {"n": 116, "acc": 0.6034, "en_acc": 0.7069, "auroc": 0.5217, "en_auroc": 0.5851, "yes": 0.2586, "en_yes": 0.2586, "gold_yes": 0.5172}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 116

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Chinese · prompt_injections · auroc

Solar Decide, same named hosted route; Chinese; slice prompt_injections. {"answered": 1979, "cases": 1979, "complete": true}

Source observations: {"n": 116, "acc": 0.6034, "en_acc": 0.7069, "auroc": 0.5217, "en_auroc": 0.5851, "yes": 0.2586, "en_yes": 0.2586, "gold_yes": 0.5172}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 116

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Chinese · prompt_injections · en_acc

Solar Decide, same named hosted route; Chinese; slice prompt_injections. {"answered": 1979, "cases": 1979, "complete": true}

Source observations: {"n": 116, "acc": 0.6034, "en_acc": 0.7069, "auroc": 0.5217, "en_auroc": 0.5851, "yes": 0.2586, "en_yes": 0.2586, "gold_yes": 0.5172}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 116

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Chinese · prompt_injections · en_auroc

Solar Decide, same named hosted route; Chinese; slice prompt_injections. {"answered": 1979, "cases": 1979, "complete": true}

Source observations: {"n": 116, "acc": 0.6034, "en_acc": 0.7069, "auroc": 0.5217, "en_auroc": 0.5851, "yes": 0.2586, "en_yes": 0.2586, "gold_yes": 0.5172}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 116

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Chinese · prompt_injections · yes

Solar Decide, same named hosted route; Chinese; slice prompt_injections. {"answered": 1979, "cases": 1979, "complete": true}

Source observations: {"n": 116, "acc": 0.6034, "en_acc": 0.7069, "auroc": 0.5217, "en_auroc": 0.5851, "yes": 0.2586, "en_yes": 0.2586, "gold_yes": 0.5172}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 116

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Chinese · prompt_injections · en_yes

Solar Decide, same named hosted route; Chinese; slice prompt_injections. {"answered": 1979, "cases": 1979, "complete": true}

Source observations: {"n": 116, "acc": 0.6034, "en_acc": 0.7069, "auroc": 0.5217, "en_auroc": 0.5851, "yes": 0.2586, "en_yes": 0.2586, "gold_yes": 0.5172}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 116

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Arabic · overall · acc

Solar Decide, same named hosted route; Arabic; slice overall. {"answered": 2007, "cases": 2007, "complete": true}

Source observations: {"n": 2007, "acc": 0.7273, "en_acc": 0.7947, "delta": -0.0675, "ci": [-0.0871, -0.0489], "significant": true, "auroc": 0.7859, "en_auroc": 0.7966}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 2007

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Arabic · overall · auroc

Solar Decide, same named hosted route; Arabic; slice overall. {"answered": 2007, "cases": 2007, "complete": true}

Source observations: {"n": 2007, "acc": 0.7273, "en_acc": 0.7947, "delta": -0.0675, "ci": [-0.0871, -0.0489], "significant": true, "auroc": 0.7859, "en_auroc": 0.7966}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 2007

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Arabic · overall · en_acc

Solar Decide, same named hosted route; Arabic; slice overall. {"answered": 2007, "cases": 2007, "complete": true}

Source observations: {"n": 2007, "acc": 0.7273, "en_acc": 0.7947, "delta": -0.0675, "ci": [-0.0871, -0.0489], "significant": true, "auroc": 0.7859, "en_auroc": 0.7966}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 2007

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Arabic · overall · en_auroc

Solar Decide, same named hosted route; Arabic; slice overall. {"answered": 2007, "cases": 2007, "complete": true}

Source observations: {"n": 2007, "acc": 0.7273, "en_acc": 0.7947, "delta": -0.0675, "ci": [-0.0871, -0.0489], "significant": true, "auroc": 0.7859, "en_auroc": 0.7966}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 2007

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Arabic · overall · delta

Solar Decide, same named hosted route; Arabic; slice overall. {"answered": 2007, "cases": 2007, "complete": true}

Source observations: {"n": 2007, "acc": 0.7273, "en_acc": 0.7947, "delta": -0.0675, "ci": [-0.0871, -0.0489], "significant": true, "auroc": 0.7859, "en_auroc": 0.7966}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 2007

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Arabic · aegis2 · acc

Solar Decide, same named hosted route; Arabic; slice aegis2. {"answered": 2007, "cases": 2007, "complete": true}

Source observations: {"n": 500, "acc": 0.806, "en_acc": 0.808, "auroc": 0.8211, "en_auroc": 0.8447, "yes": 0.386, "en_yes": 0.364, "gold_yes": 0.5}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 500

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Arabic · aegis2 · auroc

Solar Decide, same named hosted route; Arabic; slice aegis2. {"answered": 2007, "cases": 2007, "complete": true}

Source observations: {"n": 500, "acc": 0.806, "en_acc": 0.808, "auroc": 0.8211, "en_auroc": 0.8447, "yes": 0.386, "en_yes": 0.364, "gold_yes": 0.5}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 500

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Arabic · aegis2 · en_acc

Solar Decide, same named hosted route; Arabic; slice aegis2. {"answered": 2007, "cases": 2007, "complete": true}

Source observations: {"n": 500, "acc": 0.806, "en_acc": 0.808, "auroc": 0.8211, "en_auroc": 0.8447, "yes": 0.386, "en_yes": 0.364, "gold_yes": 0.5}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 500

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Arabic · aegis2 · en_auroc

Solar Decide, same named hosted route; Arabic; slice aegis2. {"answered": 2007, "cases": 2007, "complete": true}

Source observations: {"n": 500, "acc": 0.806, "en_acc": 0.808, "auroc": 0.8211, "en_auroc": 0.8447, "yes": 0.386, "en_yes": 0.364, "gold_yes": 0.5}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 500

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Arabic · aegis2 · yes

Solar Decide, same named hosted route; Arabic; slice aegis2. {"answered": 2007, "cases": 2007, "complete": true}

Source observations: {"n": 500, "acc": 0.806, "en_acc": 0.808, "auroc": 0.8211, "en_auroc": 0.8447, "yes": 0.386, "en_yes": 0.364, "gold_yes": 0.5}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 500

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Arabic · aegis2 · en_yes

Solar Decide, same named hosted route; Arabic; slice aegis2. {"answered": 2007, "cases": 2007, "complete": true}

Source observations: {"n": 500, "acc": 0.806, "en_acc": 0.808, "auroc": 0.8211, "en_auroc": 0.8447, "yes": 0.386, "en_yes": 0.364, "gold_yes": 0.5}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 500

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Arabic · aegis2_response · acc

Solar Decide, same named hosted route; Arabic; slice aegis2_response. {"answered": 2007, "cases": 2007, "complete": true}

Source observations: {"n": 499, "acc": 0.7655, "en_acc": 0.7535, "auroc": 0.8283, "en_auroc": 0.8, "yes": 0.3487, "en_yes": 0.2966, "gold_yes": 0.499}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 499

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Arabic · aegis2_response · auroc

Solar Decide, same named hosted route; Arabic; slice aegis2_response. {"answered": 2007, "cases": 2007, "complete": true}

Source observations: {"n": 499, "acc": 0.7655, "en_acc": 0.7535, "auroc": 0.8283, "en_auroc": 0.8, "yes": 0.3487, "en_yes": 0.2966, "gold_yes": 0.499}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 499

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Arabic · aegis2_response · en_acc

Solar Decide, same named hosted route; Arabic; slice aegis2_response. {"answered": 2007, "cases": 2007, "complete": true}

Source observations: {"n": 499, "acc": 0.7655, "en_acc": 0.7535, "auroc": 0.8283, "en_auroc": 0.8, "yes": 0.3487, "en_yes": 0.2966, "gold_yes": 0.499}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 499

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Arabic · aegis2_response · en_auroc

Solar Decide, same named hosted route; Arabic; slice aegis2_response. {"answered": 2007, "cases": 2007, "complete": true}

Source observations: {"n": 499, "acc": 0.7655, "en_acc": 0.7535, "auroc": 0.8283, "en_auroc": 0.8, "yes": 0.3487, "en_yes": 0.2966, "gold_yes": 0.499}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 499

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Arabic · aegis2_response · yes

Solar Decide, same named hosted route; Arabic; slice aegis2_response. {"answered": 2007, "cases": 2007, "complete": true}

Source observations: {"n": 499, "acc": 0.7655, "en_acc": 0.7535, "auroc": 0.8283, "en_auroc": 0.8, "yes": 0.3487, "en_yes": 0.2966, "gold_yes": 0.499}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 499

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Arabic · aegis2_response · en_yes

Solar Decide, same named hosted route; Arabic; slice aegis2_response. {"answered": 2007, "cases": 2007, "complete": true}

Source observations: {"n": 499, "acc": 0.7655, "en_acc": 0.7535, "auroc": 0.8283, "en_auroc": 0.8, "yes": 0.3487, "en_yes": 0.2966, "gold_yes": 0.499}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 499

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Arabic · atbench500 · acc

Solar Decide, same named hosted route; Arabic; slice atbench500. {"answered": 2007, "cases": 2007, "complete": true}

Source observations: {"n": 493, "acc": 0.5274, "en_acc": 0.7404, "auroc": 0.5793, "en_auroc": 0.8042, "yes": 0.783, "en_yes": 0.5538, "gold_yes": 0.5051}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 493

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Arabic · atbench500 · auroc

Solar Decide, same named hosted route; Arabic; slice atbench500. {"answered": 2007, "cases": 2007, "complete": true}

Source observations: {"n": 493, "acc": 0.5274, "en_acc": 0.7404, "auroc": 0.5793, "en_auroc": 0.8042, "yes": 0.783, "en_yes": 0.5538, "gold_yes": 0.5051}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 493

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Arabic · atbench500 · en_acc

Solar Decide, same named hosted route; Arabic; slice atbench500. {"answered": 2007, "cases": 2007, "complete": true}

Source observations: {"n": 493, "acc": 0.5274, "en_acc": 0.7404, "auroc": 0.5793, "en_auroc": 0.8042, "yes": 0.783, "en_yes": 0.5538, "gold_yes": 0.5051}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 493

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Arabic · atbench500 · en_auroc

Solar Decide, same named hosted route; Arabic; slice atbench500. {"answered": 2007, "cases": 2007, "complete": true}

Source observations: {"n": 493, "acc": 0.5274, "en_acc": 0.7404, "auroc": 0.5793, "en_auroc": 0.8042, "yes": 0.783, "en_yes": 0.5538, "gold_yes": 0.5051}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 493

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Arabic · atbench500 · yes

Solar Decide, same named hosted route; Arabic; slice atbench500. {"answered": 2007, "cases": 2007, "complete": true}

Source observations: {"n": 493, "acc": 0.5274, "en_acc": 0.7404, "auroc": 0.5793, "en_auroc": 0.8042, "yes": 0.783, "en_yes": 0.5538, "gold_yes": 0.5051}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 493

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Arabic · atbench500 · en_yes

Solar Decide, same named hosted route; Arabic; slice atbench500. {"answered": 2007, "cases": 2007, "complete": true}

Source observations: {"n": 493, "acc": 0.5274, "en_acc": 0.7404, "auroc": 0.5793, "en_auroc": 0.8042, "yes": 0.783, "en_yes": 0.5538, "gold_yes": 0.5051}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 493

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Arabic · jailbreak_classification · acc

Solar Decide, same named hosted route; Arabic; slice jailbreak_classification. {"answered": 2007, "cases": 2007, "complete": true}

Source observations: {"n": 399, "acc": 0.8822, "en_acc": 0.9649, "auroc": 0.9175, "en_auroc": 0.9491, "yes": 0.4386, "en_yes": 0.3258, "gold_yes": 0.3509}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 399

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Arabic · jailbreak_classification · auroc

Solar Decide, same named hosted route; Arabic; slice jailbreak_classification. {"answered": 2007, "cases": 2007, "complete": true}

Source observations: {"n": 399, "acc": 0.8822, "en_acc": 0.9649, "auroc": 0.9175, "en_auroc": 0.9491, "yes": 0.4386, "en_yes": 0.3258, "gold_yes": 0.3509}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 399

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Arabic · jailbreak_classification · en_acc

Solar Decide, same named hosted route; Arabic; slice jailbreak_classification. {"answered": 2007, "cases": 2007, "complete": true}

Source observations: {"n": 399, "acc": 0.8822, "en_acc": 0.9649, "auroc": 0.9175, "en_auroc": 0.9491, "yes": 0.4386, "en_yes": 0.3258, "gold_yes": 0.3509}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 399

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Arabic · jailbreak_classification · en_auroc

Solar Decide, same named hosted route; Arabic; slice jailbreak_classification. {"answered": 2007, "cases": 2007, "complete": true}

Source observations: {"n": 399, "acc": 0.8822, "en_acc": 0.9649, "auroc": 0.9175, "en_auroc": 0.9491, "yes": 0.4386, "en_yes": 0.3258, "gold_yes": 0.3509}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 399

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Arabic · jailbreak_classification · yes

Solar Decide, same named hosted route; Arabic; slice jailbreak_classification. {"answered": 2007, "cases": 2007, "complete": true}

Source observations: {"n": 399, "acc": 0.8822, "en_acc": 0.9649, "auroc": 0.9175, "en_auroc": 0.9491, "yes": 0.4386, "en_yes": 0.3258, "gold_yes": 0.3509}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 399

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Arabic · jailbreak_classification · en_yes

Solar Decide, same named hosted route; Arabic; slice jailbreak_classification. {"answered": 2007, "cases": 2007, "complete": true}

Source observations: {"n": 399, "acc": 0.8822, "en_acc": 0.9649, "auroc": 0.9175, "en_auroc": 0.9491, "yes": 0.4386, "en_yes": 0.3258, "gold_yes": 0.3509}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 399

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Arabic · prompt_injections · acc

Solar Decide, same named hosted route; Arabic; slice prompt_injections. {"answered": 2007, "cases": 2007, "complete": true}

Source observations: {"n": 116, "acc": 0.6552, "en_acc": 0.7069, "auroc": 0.7835, "en_auroc": 0.5851, "yes": 0.1724, "en_yes": 0.2586, "gold_yes": 0.5172}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 116

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Arabic · prompt_injections · auroc

Solar Decide, same named hosted route; Arabic; slice prompt_injections. {"answered": 2007, "cases": 2007, "complete": true}

Source observations: {"n": 116, "acc": 0.6552, "en_acc": 0.7069, "auroc": 0.7835, "en_auroc": 0.5851, "yes": 0.1724, "en_yes": 0.2586, "gold_yes": 0.5172}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 116

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Arabic · prompt_injections · en_acc

Solar Decide, same named hosted route; Arabic; slice prompt_injections. {"answered": 2007, "cases": 2007, "complete": true}

Source observations: {"n": 116, "acc": 0.6552, "en_acc": 0.7069, "auroc": 0.7835, "en_auroc": 0.5851, "yes": 0.1724, "en_yes": 0.2586, "gold_yes": 0.5172}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 116

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Arabic · prompt_injections · en_auroc

Solar Decide, same named hosted route; Arabic; slice prompt_injections. {"answered": 2007, "cases": 2007, "complete": true}

Source observations: {"n": 116, "acc": 0.6552, "en_acc": 0.7069, "auroc": 0.7835, "en_auroc": 0.5851, "yes": 0.1724, "en_yes": 0.2586, "gold_yes": 0.5172}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 116

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Arabic · prompt_injections · yes

Solar Decide, same named hosted route; Arabic; slice prompt_injections. {"answered": 2007, "cases": 2007, "complete": true}

Source observations: {"n": 116, "acc": 0.6552, "en_acc": 0.7069, "auroc": 0.7835, "en_auroc": 0.5851, "yes": 0.1724, "en_yes": 0.2586, "gold_yes": 0.5172}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 116

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Arabic · prompt_injections · en_yes

Solar Decide, same named hosted route; Arabic; slice prompt_injections. {"answered": 2007, "cases": 2007, "complete": true}

Source observations: {"n": 116, "acc": 0.6552, "en_acc": 0.7069, "auroc": 0.7835, "en_auroc": 0.5851, "yes": 0.1724, "en_yes": 0.2586, "gold_yes": 0.5172}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 116

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Hindi · overall · acc

Solar Decide, same named hosted route; Hindi; slice overall. {"answered": 1999, "cases": 1999, "complete": true}

Source observations: {"n": 1999, "acc": 0.7285, "en_acc": 0.7954, "delta": -0.0669, "ci": [-0.0864, -0.0486], "significant": true, "auroc": 0.7504, "en_auroc": 0.7978}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 1999

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Hindi · overall · auroc

Solar Decide, same named hosted route; Hindi; slice overall. {"answered": 1999, "cases": 1999, "complete": true}

Source observations: {"n": 1999, "acc": 0.7285, "en_acc": 0.7954, "delta": -0.0669, "ci": [-0.0864, -0.0486], "significant": true, "auroc": 0.7504, "en_auroc": 0.7978}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 1999

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Hindi · overall · en_acc

Solar Decide, same named hosted route; Hindi; slice overall. {"answered": 1999, "cases": 1999, "complete": true}

Source observations: {"n": 1999, "acc": 0.7285, "en_acc": 0.7954, "delta": -0.0669, "ci": [-0.0864, -0.0486], "significant": true, "auroc": 0.7504, "en_auroc": 0.7978}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 1999

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Hindi · overall · en_auroc

Solar Decide, same named hosted route; Hindi; slice overall. {"answered": 1999, "cases": 1999, "complete": true}

Source observations: {"n": 1999, "acc": 0.7285, "en_acc": 0.7954, "delta": -0.0669, "ci": [-0.0864, -0.0486], "significant": true, "auroc": 0.7504, "en_auroc": 0.7978}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 1999

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Hindi · overall · delta

Solar Decide, same named hosted route; Hindi; slice overall. {"answered": 1999, "cases": 1999, "complete": true}

Source observations: {"n": 1999, "acc": 0.7285, "en_acc": 0.7954, "delta": -0.0669, "ci": [-0.0864, -0.0486], "significant": true, "auroc": 0.7504, "en_auroc": 0.7978}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 1999

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Hindi · aegis2 · acc

Solar Decide, same named hosted route; Hindi; slice aegis2. {"answered": 1999, "cases": 1999, "complete": true}

Source observations: {"n": 500, "acc": 0.7, "en_acc": 0.808, "auroc": 0.7757, "en_auroc": 0.8447, "yes": 0.244, "en_yes": 0.364, "gold_yes": 0.5}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 500

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Hindi · aegis2 · auroc

Solar Decide, same named hosted route; Hindi; slice aegis2. {"answered": 1999, "cases": 1999, "complete": true}

Source observations: {"n": 500, "acc": 0.7, "en_acc": 0.808, "auroc": 0.7757, "en_auroc": 0.8447, "yes": 0.244, "en_yes": 0.364, "gold_yes": 0.5}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 500

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Hindi · aegis2 · en_acc

Solar Decide, same named hosted route; Hindi; slice aegis2. {"answered": 1999, "cases": 1999, "complete": true}

Source observations: {"n": 500, "acc": 0.7, "en_acc": 0.808, "auroc": 0.7757, "en_auroc": 0.8447, "yes": 0.244, "en_yes": 0.364, "gold_yes": 0.5}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 500

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Hindi · aegis2 · en_auroc

Solar Decide, same named hosted route; Hindi; slice aegis2. {"answered": 1999, "cases": 1999, "complete": true}

Source observations: {"n": 500, "acc": 0.7, "en_acc": 0.808, "auroc": 0.7757, "en_auroc": 0.8447, "yes": 0.244, "en_yes": 0.364, "gold_yes": 0.5}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 500

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Hindi · aegis2 · yes

Solar Decide, same named hosted route; Hindi; slice aegis2. {"answered": 1999, "cases": 1999, "complete": true}

Source observations: {"n": 500, "acc": 0.7, "en_acc": 0.808, "auroc": 0.7757, "en_auroc": 0.8447, "yes": 0.244, "en_yes": 0.364, "gold_yes": 0.5}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 500

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Hindi · aegis2 · en_yes

Solar Decide, same named hosted route; Hindi; slice aegis2. {"answered": 1999, "cases": 1999, "complete": true}

Source observations: {"n": 500, "acc": 0.7, "en_acc": 0.808, "auroc": 0.7757, "en_auroc": 0.8447, "yes": 0.244, "en_yes": 0.364, "gold_yes": 0.5}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 500

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Hindi · aegis2_response · acc

Solar Decide, same named hosted route; Hindi; slice aegis2_response. {"answered": 1999, "cases": 1999, "complete": true}

Source observations: {"n": 499, "acc": 0.7475, "en_acc": 0.7535, "auroc": 0.8231, "en_auroc": 0.8, "yes": 0.3226, "en_yes": 0.2966, "gold_yes": 0.499}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 499

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Hindi · aegis2_response · auroc

Solar Decide, same named hosted route; Hindi; slice aegis2_response. {"answered": 1999, "cases": 1999, "complete": true}

Source observations: {"n": 499, "acc": 0.7475, "en_acc": 0.7535, "auroc": 0.8231, "en_auroc": 0.8, "yes": 0.3226, "en_yes": 0.2966, "gold_yes": 0.499}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 499

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Hindi · aegis2_response · en_acc

Solar Decide, same named hosted route; Hindi; slice aegis2_response. {"answered": 1999, "cases": 1999, "complete": true}

Source observations: {"n": 499, "acc": 0.7475, "en_acc": 0.7535, "auroc": 0.8231, "en_auroc": 0.8, "yes": 0.3226, "en_yes": 0.2966, "gold_yes": 0.499}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 499

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Hindi · aegis2_response · en_auroc

Solar Decide, same named hosted route; Hindi; slice aegis2_response. {"answered": 1999, "cases": 1999, "complete": true}

Source observations: {"n": 499, "acc": 0.7475, "en_acc": 0.7535, "auroc": 0.8231, "en_auroc": 0.8, "yes": 0.3226, "en_yes": 0.2966, "gold_yes": 0.499}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 499

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Hindi · aegis2_response · yes

Solar Decide, same named hosted route; Hindi; slice aegis2_response. {"answered": 1999, "cases": 1999, "complete": true}

Source observations: {"n": 499, "acc": 0.7475, "en_acc": 0.7535, "auroc": 0.8231, "en_auroc": 0.8, "yes": 0.3226, "en_yes": 0.2966, "gold_yes": 0.499}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 499

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Hindi · aegis2_response · en_yes

Solar Decide, same named hosted route; Hindi; slice aegis2_response. {"answered": 1999, "cases": 1999, "complete": true}

Source observations: {"n": 499, "acc": 0.7475, "en_acc": 0.7535, "auroc": 0.8231, "en_auroc": 0.8, "yes": 0.3226, "en_yes": 0.2966, "gold_yes": 0.499}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 499

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Hindi · atbench500 · acc

Solar Decide, same named hosted route; Hindi; slice atbench500. {"answered": 1999, "cases": 1999, "complete": true}

Source observations: {"n": 487, "acc": 0.5852, "en_acc": 0.7413, "auroc": 0.6329, "en_auroc": 0.8041, "yes": 0.731, "en_yes": 0.5544, "gold_yes": 0.5051}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 487

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Hindi · atbench500 · auroc

Solar Decide, same named hosted route; Hindi; slice atbench500. {"answered": 1999, "cases": 1999, "complete": true}

Source observations: {"n": 487, "acc": 0.5852, "en_acc": 0.7413, "auroc": 0.6329, "en_auroc": 0.8041, "yes": 0.731, "en_yes": 0.5544, "gold_yes": 0.5051}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 487

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Hindi · atbench500 · en_acc

Solar Decide, same named hosted route; Hindi; slice atbench500. {"answered": 1999, "cases": 1999, "complete": true}

Source observations: {"n": 487, "acc": 0.5852, "en_acc": 0.7413, "auroc": 0.6329, "en_auroc": 0.8041, "yes": 0.731, "en_yes": 0.5544, "gold_yes": 0.5051}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 487

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Hindi · atbench500 · en_auroc

Solar Decide, same named hosted route; Hindi; slice atbench500. {"answered": 1999, "cases": 1999, "complete": true}

Source observations: {"n": 487, "acc": 0.5852, "en_acc": 0.7413, "auroc": 0.6329, "en_auroc": 0.8041, "yes": 0.731, "en_yes": 0.5544, "gold_yes": 0.5051}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 487

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Hindi · atbench500 · yes

Solar Decide, same named hosted route; Hindi; slice atbench500. {"answered": 1999, "cases": 1999, "complete": true}

Source observations: {"n": 487, "acc": 0.5852, "en_acc": 0.7413, "auroc": 0.6329, "en_auroc": 0.8041, "yes": 0.731, "en_yes": 0.5544, "gold_yes": 0.5051}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 487

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Hindi · atbench500 · en_yes

Solar Decide, same named hosted route; Hindi; slice atbench500. {"answered": 1999, "cases": 1999, "complete": true}

Source observations: {"n": 487, "acc": 0.5852, "en_acc": 0.7413, "auroc": 0.6329, "en_auroc": 0.8041, "yes": 0.731, "en_yes": 0.5544, "gold_yes": 0.5051}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 487

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Hindi · jailbreak_classification · acc

Solar Decide, same named hosted route; Hindi; slice jailbreak_classification. {"answered": 1999, "cases": 1999, "complete": true}

Source observations: {"n": 397, "acc": 0.9547, "en_acc": 0.9673, "auroc": 0.9476, "en_auroc": 0.9552, "yes": 0.3224, "en_yes": 0.3249, "gold_yes": 0.3476}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 397

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Hindi · jailbreak_classification · auroc

Solar Decide, same named hosted route; Hindi; slice jailbreak_classification. {"answered": 1999, "cases": 1999, "complete": true}

Source observations: {"n": 397, "acc": 0.9547, "en_acc": 0.9673, "auroc": 0.9476, "en_auroc": 0.9552, "yes": 0.3224, "en_yes": 0.3249, "gold_yes": 0.3476}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 397

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Hindi · jailbreak_classification · en_acc

Solar Decide, same named hosted route; Hindi; slice jailbreak_classification. {"answered": 1999, "cases": 1999, "complete": true}

Source observations: {"n": 397, "acc": 0.9547, "en_acc": 0.9673, "auroc": 0.9476, "en_auroc": 0.9552, "yes": 0.3224, "en_yes": 0.3249, "gold_yes": 0.3476}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 397

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Hindi · jailbreak_classification · en_auroc

Solar Decide, same named hosted route; Hindi; slice jailbreak_classification. {"answered": 1999, "cases": 1999, "complete": true}

Source observations: {"n": 397, "acc": 0.9547, "en_acc": 0.9673, "auroc": 0.9476, "en_auroc": 0.9552, "yes": 0.3224, "en_yes": 0.3249, "gold_yes": 0.3476}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 397

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Hindi · jailbreak_classification · yes

Solar Decide, same named hosted route; Hindi; slice jailbreak_classification. {"answered": 1999, "cases": 1999, "complete": true}

Source observations: {"n": 397, "acc": 0.9547, "en_acc": 0.9673, "auroc": 0.9476, "en_auroc": 0.9552, "yes": 0.3224, "en_yes": 0.3249, "gold_yes": 0.3476}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 397

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Hindi · jailbreak_classification · en_yes

Solar Decide, same named hosted route; Hindi; slice jailbreak_classification. {"answered": 1999, "cases": 1999, "complete": true}

Source observations: {"n": 397, "acc": 0.9547, "en_acc": 0.9673, "auroc": 0.9476, "en_auroc": 0.9552, "yes": 0.3224, "en_yes": 0.3249, "gold_yes": 0.3476}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 397

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Hindi · prompt_injections · acc

Solar Decide, same named hosted route; Hindi; slice prompt_injections. {"answered": 1999, "cases": 1999, "complete": true}

Source observations: {"n": 116, "acc": 0.6552, "en_acc": 0.7069, "auroc": 0.5729, "en_auroc": 0.5851, "yes": 0.1724, "en_yes": 0.2586, "gold_yes": 0.5172}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 116

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Hindi · prompt_injections · auroc

Solar Decide, same named hosted route; Hindi; slice prompt_injections. {"answered": 1999, "cases": 1999, "complete": true}

Source observations: {"n": 116, "acc": 0.6552, "en_acc": 0.7069, "auroc": 0.5729, "en_auroc": 0.5851, "yes": 0.1724, "en_yes": 0.2586, "gold_yes": 0.5172}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 116

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Hindi · prompt_injections · en_acc

Solar Decide, same named hosted route; Hindi; slice prompt_injections. {"answered": 1999, "cases": 1999, "complete": true}

Source observations: {"n": 116, "acc": 0.6552, "en_acc": 0.7069, "auroc": 0.5729, "en_auroc": 0.5851, "yes": 0.1724, "en_yes": 0.2586, "gold_yes": 0.5172}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 116

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Hindi · prompt_injections · en_auroc

Solar Decide, same named hosted route; Hindi; slice prompt_injections. {"answered": 1999, "cases": 1999, "complete": true}

Source observations: {"n": 116, "acc": 0.6552, "en_acc": 0.7069, "auroc": 0.5729, "en_auroc": 0.5851, "yes": 0.1724, "en_yes": 0.2586, "gold_yes": 0.5172}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 116

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Hindi · prompt_injections · yes

Solar Decide, same named hosted route; Hindi; slice prompt_injections. {"answered": 1999, "cases": 1999, "complete": true}

Source observations: {"n": 116, "acc": 0.6552, "en_acc": 0.7069, "auroc": 0.5729, "en_auroc": 0.5851, "yes": 0.1724, "en_yes": 0.2586, "gold_yes": 0.5172}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 116

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Hindi · prompt_injections · en_yes

Solar Decide, same named hosted route; Hindi; slice prompt_injections. {"answered": 1999, "cases": 1999, "complete": true}

Source observations: {"n": 116, "acc": 0.6552, "en_acc": 0.7069, "auroc": 0.5729, "en_auroc": 0.5851, "yes": 0.1724, "en_yes": 0.2586, "gold_yes": 0.5172}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 116

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Vietnamese · overall · acc

Solar Decide, same named hosted route; Vietnamese; slice overall. {"answered": 1968, "cases": 1968, "complete": true}

Source observations: {"n": 1968, "acc": 0.7052, "en_acc": 0.7958, "delta": -0.0906, "ci": [-0.1141, -0.0677], "significant": true, "auroc": 0.7144, "en_auroc": 0.7986}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 1968

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Vietnamese · overall · auroc

Solar Decide, same named hosted route; Vietnamese; slice overall. {"answered": 1968, "cases": 1968, "complete": true}

Source observations: {"n": 1968, "acc": 0.7052, "en_acc": 0.7958, "delta": -0.0906, "ci": [-0.1141, -0.0677], "significant": true, "auroc": 0.7144, "en_auroc": 0.7986}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 1968

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Vietnamese · overall · en_acc

Solar Decide, same named hosted route; Vietnamese; slice overall. {"answered": 1968, "cases": 1968, "complete": true}

Source observations: {"n": 1968, "acc": 0.7052, "en_acc": 0.7958, "delta": -0.0906, "ci": [-0.1141, -0.0677], "significant": true, "auroc": 0.7144, "en_auroc": 0.7986}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 1968

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Vietnamese · overall · en_auroc

Solar Decide, same named hosted route; Vietnamese; slice overall. {"answered": 1968, "cases": 1968, "complete": true}

Source observations: {"n": 1968, "acc": 0.7052, "en_acc": 0.7958, "delta": -0.0906, "ci": [-0.1141, -0.0677], "significant": true, "auroc": 0.7144, "en_auroc": 0.7986}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 1968

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Vietnamese · overall · delta

Solar Decide, same named hosted route; Vietnamese; slice overall. {"answered": 1968, "cases": 1968, "complete": true}

Source observations: {"n": 1968, "acc": 0.7052, "en_acc": 0.7958, "delta": -0.0906, "ci": [-0.1141, -0.0677], "significant": true, "auroc": 0.7144, "en_auroc": 0.7986}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 1968

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Vietnamese · aegis2 · acc

Solar Decide, same named hosted route; Vietnamese; slice aegis2. {"answered": 1968, "cases": 1968, "complete": true}

Source observations: {"n": 497, "acc": 0.7666, "en_acc": 0.8068, "auroc": 0.8378, "en_auroc": 0.8429, "yes": 0.336, "en_yes": 0.3602, "gold_yes": 0.497}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 497

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Vietnamese · aegis2 · auroc

Solar Decide, same named hosted route; Vietnamese; slice aegis2. {"answered": 1968, "cases": 1968, "complete": true}

Source observations: {"n": 497, "acc": 0.7666, "en_acc": 0.8068, "auroc": 0.8378, "en_auroc": 0.8429, "yes": 0.336, "en_yes": 0.3602, "gold_yes": 0.497}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 497

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Vietnamese · aegis2 · en_acc

Solar Decide, same named hosted route; Vietnamese; slice aegis2. {"answered": 1968, "cases": 1968, "complete": true}

Source observations: {"n": 497, "acc": 0.7666, "en_acc": 0.8068, "auroc": 0.8378, "en_auroc": 0.8429, "yes": 0.336, "en_yes": 0.3602, "gold_yes": 0.497}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 497

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Vietnamese · aegis2 · en_auroc

Solar Decide, same named hosted route; Vietnamese; slice aegis2. {"answered": 1968, "cases": 1968, "complete": true}

Source observations: {"n": 497, "acc": 0.7666, "en_acc": 0.8068, "auroc": 0.8378, "en_auroc": 0.8429, "yes": 0.336, "en_yes": 0.3602, "gold_yes": 0.497}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 497

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Vietnamese · aegis2 · yes

Solar Decide, same named hosted route; Vietnamese; slice aegis2. {"answered": 1968, "cases": 1968, "complete": true}

Source observations: {"n": 497, "acc": 0.7666, "en_acc": 0.8068, "auroc": 0.8378, "en_auroc": 0.8429, "yes": 0.336, "en_yes": 0.3602, "gold_yes": 0.497}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 497

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Vietnamese · aegis2 · en_yes

Solar Decide, same named hosted route; Vietnamese; slice aegis2. {"answered": 1968, "cases": 1968, "complete": true}

Source observations: {"n": 497, "acc": 0.7666, "en_acc": 0.8068, "auroc": 0.8378, "en_auroc": 0.8429, "yes": 0.336, "en_yes": 0.3602, "gold_yes": 0.497}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 497

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Vietnamese · aegis2_response · acc

Solar Decide, same named hosted route; Vietnamese; slice aegis2_response. {"answered": 1968, "cases": 1968, "complete": true}

Source observations: {"n": 499, "acc": 0.7355, "en_acc": 0.7555, "auroc": 0.7523, "en_auroc": 0.8028, "yes": 0.3106, "en_yes": 0.2986, "gold_yes": 0.499}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 499

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Vietnamese · aegis2_response · auroc

Solar Decide, same named hosted route; Vietnamese; slice aegis2_response. {"answered": 1968, "cases": 1968, "complete": true}

Source observations: {"n": 499, "acc": 0.7355, "en_acc": 0.7555, "auroc": 0.7523, "en_auroc": 0.8028, "yes": 0.3106, "en_yes": 0.2986, "gold_yes": 0.499}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 499

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Vietnamese · aegis2_response · en_acc

Solar Decide, same named hosted route; Vietnamese; slice aegis2_response. {"answered": 1968, "cases": 1968, "complete": true}

Source observations: {"n": 499, "acc": 0.7355, "en_acc": 0.7555, "auroc": 0.7523, "en_auroc": 0.8028, "yes": 0.3106, "en_yes": 0.2986, "gold_yes": 0.499}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 499

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Vietnamese · aegis2_response · en_auroc

Solar Decide, same named hosted route; Vietnamese; slice aegis2_response. {"answered": 1968, "cases": 1968, "complete": true}

Source observations: {"n": 499, "acc": 0.7355, "en_acc": 0.7555, "auroc": 0.7523, "en_auroc": 0.8028, "yes": 0.3106, "en_yes": 0.2986, "gold_yes": 0.499}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 499

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Vietnamese · aegis2_response · yes

Solar Decide, same named hosted route; Vietnamese; slice aegis2_response. {"answered": 1968, "cases": 1968, "complete": true}

Source observations: {"n": 499, "acc": 0.7355, "en_acc": 0.7555, "auroc": 0.7523, "en_auroc": 0.8028, "yes": 0.3106, "en_yes": 0.2986, "gold_yes": 0.499}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 499

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Vietnamese · aegis2_response · en_yes

Solar Decide, same named hosted route; Vietnamese; slice aegis2_response. {"answered": 1968, "cases": 1968, "complete": true}

Source observations: {"n": 499, "acc": 0.7355, "en_acc": 0.7555, "auroc": 0.7523, "en_auroc": 0.8028, "yes": 0.3106, "en_yes": 0.2986, "gold_yes": 0.499}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 499

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Vietnamese · atbench500 · acc

Solar Decide, same named hosted route; Vietnamese; slice atbench500. {"answered": 1968, "cases": 1968, "complete": true}

Source observations: {"n": 463, "acc": 0.5205, "en_acc": 0.7365, "auroc": 0.575, "en_auroc": 0.7984, "yes": 0.7797, "en_yes": 0.5637, "gold_yes": 0.5119}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 463

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Vietnamese · atbench500 · auroc

Solar Decide, same named hosted route; Vietnamese; slice atbench500. {"answered": 1968, "cases": 1968, "complete": true}

Source observations: {"n": 463, "acc": 0.5205, "en_acc": 0.7365, "auroc": 0.575, "en_auroc": 0.7984, "yes": 0.7797, "en_yes": 0.5637, "gold_yes": 0.5119}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 463

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Vietnamese · atbench500 · en_acc

Solar Decide, same named hosted route; Vietnamese; slice atbench500. {"answered": 1968, "cases": 1968, "complete": true}

Source observations: {"n": 463, "acc": 0.5205, "en_acc": 0.7365, "auroc": 0.575, "en_auroc": 0.7984, "yes": 0.7797, "en_yes": 0.5637, "gold_yes": 0.5119}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 463

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Vietnamese · atbench500 · en_auroc

Solar Decide, same named hosted route; Vietnamese; slice atbench500. {"answered": 1968, "cases": 1968, "complete": true}

Source observations: {"n": 463, "acc": 0.5205, "en_acc": 0.7365, "auroc": 0.575, "en_auroc": 0.7984, "yes": 0.7797, "en_yes": 0.5637, "gold_yes": 0.5119}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 463

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Vietnamese · atbench500 · yes

Solar Decide, same named hosted route; Vietnamese; slice atbench500. {"answered": 1968, "cases": 1968, "complete": true}

Source observations: {"n": 463, "acc": 0.5205, "en_acc": 0.7365, "auroc": 0.575, "en_auroc": 0.7984, "yes": 0.7797, "en_yes": 0.5637, "gold_yes": 0.5119}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 463

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Vietnamese · atbench500 · en_yes

Solar Decide, same named hosted route; Vietnamese; slice atbench500. {"answered": 1968, "cases": 1968, "complete": true}

Source observations: {"n": 463, "acc": 0.5205, "en_acc": 0.7365, "auroc": 0.575, "en_auroc": 0.7984, "yes": 0.7797, "en_yes": 0.5637, "gold_yes": 0.5119}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 463

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Vietnamese · jailbreak_classification · acc

Solar Decide, same named hosted route; Vietnamese; slice jailbreak_classification. {"answered": 1968, "cases": 1968, "complete": true}

Source observations: {"n": 394, "acc": 0.9036, "en_acc": 0.967, "auroc": 0.9104, "en_auroc": 0.9551, "yes": 0.401, "en_yes": 0.3274, "gold_yes": 0.3503}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 394

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Vietnamese · jailbreak_classification · auroc

Solar Decide, same named hosted route; Vietnamese; slice jailbreak_classification. {"answered": 1968, "cases": 1968, "complete": true}

Source observations: {"n": 394, "acc": 0.9036, "en_acc": 0.967, "auroc": 0.9104, "en_auroc": 0.9551, "yes": 0.401, "en_yes": 0.3274, "gold_yes": 0.3503}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 394

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Vietnamese · jailbreak_classification · en_acc

Solar Decide, same named hosted route; Vietnamese; slice jailbreak_classification. {"answered": 1968, "cases": 1968, "complete": true}

Source observations: {"n": 394, "acc": 0.9036, "en_acc": 0.967, "auroc": 0.9104, "en_auroc": 0.9551, "yes": 0.401, "en_yes": 0.3274, "gold_yes": 0.3503}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 394

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Vietnamese · jailbreak_classification · en_auroc

Solar Decide, same named hosted route; Vietnamese; slice jailbreak_classification. {"answered": 1968, "cases": 1968, "complete": true}

Source observations: {"n": 394, "acc": 0.9036, "en_acc": 0.967, "auroc": 0.9104, "en_auroc": 0.9551, "yes": 0.401, "en_yes": 0.3274, "gold_yes": 0.3503}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 394

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Vietnamese · jailbreak_classification · yes

Solar Decide, same named hosted route; Vietnamese; slice jailbreak_classification. {"answered": 1968, "cases": 1968, "complete": true}

Source observations: {"n": 394, "acc": 0.9036, "en_acc": 0.967, "auroc": 0.9104, "en_auroc": 0.9551, "yes": 0.401, "en_yes": 0.3274, "gold_yes": 0.3503}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 394

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Vietnamese · jailbreak_classification · en_yes

Solar Decide, same named hosted route; Vietnamese; slice jailbreak_classification. {"answered": 1968, "cases": 1968, "complete": true}

Source observations: {"n": 394, "acc": 0.9036, "en_acc": 0.967, "auroc": 0.9104, "en_auroc": 0.9551, "yes": 0.401, "en_yes": 0.3274, "gold_yes": 0.3503}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 394

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Vietnamese · prompt_injections · acc

Solar Decide, same named hosted route; Vietnamese; slice prompt_injections. {"answered": 1968, "cases": 1968, "complete": true}

Source observations: {"n": 115, "acc": 0.6, "en_acc": 0.713, "auroc": 0.4964, "en_auroc": 0.5938, "yes": 0.4087, "en_yes": 0.2609, "gold_yes": 0.513}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 115

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Vietnamese · prompt_injections · auroc

Solar Decide, same named hosted route; Vietnamese; slice prompt_injections. {"answered": 1968, "cases": 1968, "complete": true}

Source observations: {"n": 115, "acc": 0.6, "en_acc": 0.713, "auroc": 0.4964, "en_auroc": 0.5938, "yes": 0.4087, "en_yes": 0.2609, "gold_yes": 0.513}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 115

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Vietnamese · prompt_injections · en_acc

Solar Decide, same named hosted route; Vietnamese; slice prompt_injections. {"answered": 1968, "cases": 1968, "complete": true}

Source observations: {"n": 115, "acc": 0.6, "en_acc": 0.713, "auroc": 0.4964, "en_auroc": 0.5938, "yes": 0.4087, "en_yes": 0.2609, "gold_yes": 0.513}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 115

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Vietnamese · prompt_injections · en_auroc

Solar Decide, same named hosted route; Vietnamese; slice prompt_injections. {"answered": 1968, "cases": 1968, "complete": true}

Source observations: {"n": 115, "acc": 0.6, "en_acc": 0.713, "auroc": 0.4964, "en_auroc": 0.5938, "yes": 0.4087, "en_yes": 0.2609, "gold_yes": 0.513}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 115

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Vietnamese · prompt_injections · yes

Solar Decide, same named hosted route; Vietnamese; slice prompt_injections. {"answered": 1968, "cases": 1968, "complete": true}

Source observations: {"n": 115, "acc": 0.6, "en_acc": 0.713, "auroc": 0.4964, "en_auroc": 0.5938, "yes": 0.4087, "en_yes": 0.2609, "gold_yes": 0.513}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 115

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Vietnamese · prompt_injections · en_yes

Solar Decide, same named hosted route; Vietnamese; slice prompt_injections. {"answered": 1968, "cases": 1968, "complete": true}

Source observations: {"n": 115, "acc": 0.6, "en_acc": 0.713, "auroc": 0.4964, "en_auroc": 0.5938, "yes": 0.4087, "en_yes": 0.2609, "gold_yes": 0.513}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 115

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Swahili · overall · acc

Solar Decide, same named hosted route; Swahili; slice overall. {"answered": 1997, "cases": 1997, "complete": true}

Source observations: {"n": 1997, "acc": 0.6747, "en_acc": 0.7946, "delta": -0.1199, "ci": [-0.1431, -0.0989], "significant": true, "auroc": 0.7994, "en_auroc": 0.7967}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 1997

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Swahili · overall · auroc

Solar Decide, same named hosted route; Swahili; slice overall. {"answered": 1997, "cases": 1997, "complete": true}

Source observations: {"n": 1997, "acc": 0.6747, "en_acc": 0.7946, "delta": -0.1199, "ci": [-0.1431, -0.0989], "significant": true, "auroc": 0.7994, "en_auroc": 0.7967}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 1997

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Swahili · overall · en_acc

Solar Decide, same named hosted route; Swahili; slice overall. {"answered": 1997, "cases": 1997, "complete": true}

Source observations: {"n": 1997, "acc": 0.6747, "en_acc": 0.7946, "delta": -0.1199, "ci": [-0.1431, -0.0989], "significant": true, "auroc": 0.7994, "en_auroc": 0.7967}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 1997

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Swahili · overall · en_auroc

Solar Decide, same named hosted route; Swahili; slice overall. {"answered": 1997, "cases": 1997, "complete": true}

Source observations: {"n": 1997, "acc": 0.6747, "en_acc": 0.7946, "delta": -0.1199, "ci": [-0.1431, -0.0989], "significant": true, "auroc": 0.7994, "en_auroc": 0.7967}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 1997

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Swahili · overall · delta

Solar Decide, same named hosted route; Swahili; slice overall. {"answered": 1997, "cases": 1997, "complete": true}

Source observations: {"n": 1997, "acc": 0.6747, "en_acc": 0.7946, "delta": -0.1199, "ci": [-0.1431, -0.0989], "significant": true, "auroc": 0.7994, "en_auroc": 0.7967}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 1997

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Swahili · aegis2 · acc

Solar Decide, same named hosted route; Swahili; slice aegis2. {"answered": 1997, "cases": 1997, "complete": true}

Source observations: {"n": 500, "acc": 0.612, "en_acc": 0.808, "auroc": 0.6823, "en_auroc": 0.8447, "yes": 0.14, "en_yes": 0.364, "gold_yes": 0.5}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 500

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Swahili · aegis2 · auroc

Solar Decide, same named hosted route; Swahili; slice aegis2. {"answered": 1997, "cases": 1997, "complete": true}

Source observations: {"n": 500, "acc": 0.612, "en_acc": 0.808, "auroc": 0.6823, "en_auroc": 0.8447, "yes": 0.14, "en_yes": 0.364, "gold_yes": 0.5}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 500

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Swahili · aegis2 · en_acc

Solar Decide, same named hosted route; Swahili; slice aegis2. {"answered": 1997, "cases": 1997, "complete": true}

Source observations: {"n": 500, "acc": 0.612, "en_acc": 0.808, "auroc": 0.6823, "en_auroc": 0.8447, "yes": 0.14, "en_yes": 0.364, "gold_yes": 0.5}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 500

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Swahili · aegis2 · en_auroc

Solar Decide, same named hosted route; Swahili; slice aegis2. {"answered": 1997, "cases": 1997, "complete": true}

Source observations: {"n": 500, "acc": 0.612, "en_acc": 0.808, "auroc": 0.6823, "en_auroc": 0.8447, "yes": 0.14, "en_yes": 0.364, "gold_yes": 0.5}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 500

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Swahili · aegis2 · yes

Solar Decide, same named hosted route; Swahili; slice aegis2. {"answered": 1997, "cases": 1997, "complete": true}

Source observations: {"n": 500, "acc": 0.612, "en_acc": 0.808, "auroc": 0.6823, "en_auroc": 0.8447, "yes": 0.14, "en_yes": 0.364, "gold_yes": 0.5}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 500

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Swahili · aegis2 · en_yes

Solar Decide, same named hosted route; Swahili; slice aegis2. {"answered": 1997, "cases": 1997, "complete": true}

Source observations: {"n": 500, "acc": 0.612, "en_acc": 0.808, "auroc": 0.6823, "en_auroc": 0.8447, "yes": 0.14, "en_yes": 0.364, "gold_yes": 0.5}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 500

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Swahili · aegis2_response · acc

Solar Decide, same named hosted route; Swahili; slice aegis2_response. {"answered": 1997, "cases": 1997, "complete": true}

Source observations: {"n": 499, "acc": 0.6293, "en_acc": 0.7535, "auroc": 0.6963, "en_auroc": 0.8, "yes": 0.2325, "en_yes": 0.2966, "gold_yes": 0.499}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 499

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Swahili · aegis2_response · auroc

Solar Decide, same named hosted route; Swahili; slice aegis2_response. {"answered": 1997, "cases": 1997, "complete": true}

Source observations: {"n": 499, "acc": 0.6293, "en_acc": 0.7535, "auroc": 0.6963, "en_auroc": 0.8, "yes": 0.2325, "en_yes": 0.2966, "gold_yes": 0.499}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 499

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Swahili · aegis2_response · en_acc

Solar Decide, same named hosted route; Swahili; slice aegis2_response. {"answered": 1997, "cases": 1997, "complete": true}

Source observations: {"n": 499, "acc": 0.6293, "en_acc": 0.7535, "auroc": 0.6963, "en_auroc": 0.8, "yes": 0.2325, "en_yes": 0.2966, "gold_yes": 0.499}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 499

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Swahili · aegis2_response · en_auroc

Solar Decide, same named hosted route; Swahili; slice aegis2_response. {"answered": 1997, "cases": 1997, "complete": true}

Source observations: {"n": 499, "acc": 0.6293, "en_acc": 0.7535, "auroc": 0.6963, "en_auroc": 0.8, "yes": 0.2325, "en_yes": 0.2966, "gold_yes": 0.499}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 499

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Swahili · aegis2_response · yes

Solar Decide, same named hosted route; Swahili; slice aegis2_response. {"answered": 1997, "cases": 1997, "complete": true}

Source observations: {"n": 499, "acc": 0.6293, "en_acc": 0.7535, "auroc": 0.6963, "en_auroc": 0.8, "yes": 0.2325, "en_yes": 0.2966, "gold_yes": 0.499}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 499

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Swahili · aegis2_response · en_yes

Solar Decide, same named hosted route; Swahili; slice aegis2_response. {"answered": 1997, "cases": 1997, "complete": true}

Source observations: {"n": 499, "acc": 0.6293, "en_acc": 0.7535, "auroc": 0.6963, "en_auroc": 0.8, "yes": 0.2325, "en_yes": 0.2966, "gold_yes": 0.499}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 499

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Swahili · atbench500 · acc

Solar Decide, same named hosted route; Swahili; slice atbench500. {"answered": 1997, "cases": 1997, "complete": true}

Source observations: {"n": 484, "acc": 0.6281, "en_acc": 0.7397, "auroc": 0.8921, "en_auroc": 0.8049, "yes": 0.2045, "en_yes": 0.5517, "gold_yes": 0.5021}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 484

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Swahili · atbench500 · auroc

Solar Decide, same named hosted route; Swahili; slice atbench500. {"answered": 1997, "cases": 1997, "complete": true}

Source observations: {"n": 484, "acc": 0.6281, "en_acc": 0.7397, "auroc": 0.8921, "en_auroc": 0.8049, "yes": 0.2045, "en_yes": 0.5517, "gold_yes": 0.5021}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 484

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Swahili · atbench500 · en_acc

Solar Decide, same named hosted route; Swahili; slice atbench500. {"answered": 1997, "cases": 1997, "complete": true}

Source observations: {"n": 484, "acc": 0.6281, "en_acc": 0.7397, "auroc": 0.8921, "en_auroc": 0.8049, "yes": 0.2045, "en_yes": 0.5517, "gold_yes": 0.5021}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 484

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Swahili · atbench500 · en_auroc

Solar Decide, same named hosted route; Swahili; slice atbench500. {"answered": 1997, "cases": 1997, "complete": true}

Source observations: {"n": 484, "acc": 0.6281, "en_acc": 0.7397, "auroc": 0.8921, "en_auroc": 0.8049, "yes": 0.2045, "en_yes": 0.5517, "gold_yes": 0.5021}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 484

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Swahili · atbench500 · yes

Solar Decide, same named hosted route; Swahili; slice atbench500. {"answered": 1997, "cases": 1997, "complete": true}

Source observations: {"n": 484, "acc": 0.6281, "en_acc": 0.7397, "auroc": 0.8921, "en_auroc": 0.8049, "yes": 0.2045, "en_yes": 0.5517, "gold_yes": 0.5021}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 484

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Swahili · atbench500 · en_yes

Solar Decide, same named hosted route; Swahili; slice atbench500. {"answered": 1997, "cases": 1997, "complete": true}

Source observations: {"n": 484, "acc": 0.6281, "en_acc": 0.7397, "auroc": 0.8921, "en_auroc": 0.8049, "yes": 0.2045, "en_yes": 0.5517, "gold_yes": 0.5021}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 484

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Swahili · jailbreak_classification · acc

Solar Decide, same named hosted route; Swahili; slice jailbreak_classification. {"answered": 1997, "cases": 1997, "complete": true}

Source observations: {"n": 398, "acc": 0.892, "en_acc": 0.9648, "auroc": 0.9355, "en_auroc": 0.9488, "yes": 0.2714, "en_yes": 0.3241, "gold_yes": 0.3492}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 398

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Swahili · jailbreak_classification · auroc

Solar Decide, same named hosted route; Swahili; slice jailbreak_classification. {"answered": 1997, "cases": 1997, "complete": true}

Source observations: {"n": 398, "acc": 0.892, "en_acc": 0.9648, "auroc": 0.9355, "en_auroc": 0.9488, "yes": 0.2714, "en_yes": 0.3241, "gold_yes": 0.3492}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 398

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Swahili · jailbreak_classification · en_acc

Solar Decide, same named hosted route; Swahili; slice jailbreak_classification. {"answered": 1997, "cases": 1997, "complete": true}

Source observations: {"n": 398, "acc": 0.892, "en_acc": 0.9648, "auroc": 0.9355, "en_auroc": 0.9488, "yes": 0.2714, "en_yes": 0.3241, "gold_yes": 0.3492}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 398

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Swahili · jailbreak_classification · en_auroc

Solar Decide, same named hosted route; Swahili; slice jailbreak_classification. {"answered": 1997, "cases": 1997, "complete": true}

Source observations: {"n": 398, "acc": 0.892, "en_acc": 0.9648, "auroc": 0.9355, "en_auroc": 0.9488, "yes": 0.2714, "en_yes": 0.3241, "gold_yes": 0.3492}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 398

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Swahili · jailbreak_classification · yes

Solar Decide, same named hosted route; Swahili; slice jailbreak_classification. {"answered": 1997, "cases": 1997, "complete": true}

Source observations: {"n": 398, "acc": 0.892, "en_acc": 0.9648, "auroc": 0.9355, "en_auroc": 0.9488, "yes": 0.2714, "en_yes": 0.3241, "gold_yes": 0.3492}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 398

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Swahili · jailbreak_classification · en_yes

Solar Decide, same named hosted route; Swahili; slice jailbreak_classification. {"answered": 1997, "cases": 1997, "complete": true}

Source observations: {"n": 398, "acc": 0.892, "en_acc": 0.9648, "auroc": 0.9355, "en_auroc": 0.9488, "yes": 0.2714, "en_yes": 0.3241, "gold_yes": 0.3492}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 398

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Swahili · prompt_injections · acc

Solar Decide, same named hosted route; Swahili; slice prompt_injections. {"answered": 1997, "cases": 1997, "complete": true}

Source observations: {"n": 116, "acc": 0.6121, "en_acc": 0.7069, "auroc": 0.7908, "en_auroc": 0.5851, "yes": 0.1293, "en_yes": 0.2586, "gold_yes": 0.5172}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 116

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Swahili · prompt_injections · auroc

Solar Decide, same named hosted route; Swahili; slice prompt_injections. {"answered": 1997, "cases": 1997, "complete": true}

Source observations: {"n": 116, "acc": 0.6121, "en_acc": 0.7069, "auroc": 0.7908, "en_auroc": 0.5851, "yes": 0.1293, "en_yes": 0.2586, "gold_yes": 0.5172}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 116

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Swahili · prompt_injections · en_acc

Solar Decide, same named hosted route; Swahili; slice prompt_injections. {"answered": 1997, "cases": 1997, "complete": true}

Source observations: {"n": 116, "acc": 0.6121, "en_acc": 0.7069, "auroc": 0.7908, "en_auroc": 0.5851, "yes": 0.1293, "en_yes": 0.2586, "gold_yes": 0.5172}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 116

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Swahili · prompt_injections · en_auroc

Solar Decide, same named hosted route; Swahili; slice prompt_injections. {"answered": 1997, "cases": 1997, "complete": true}

Source observations: {"n": 116, "acc": 0.6121, "en_acc": 0.7069, "auroc": 0.7908, "en_auroc": 0.5851, "yes": 0.1293, "en_yes": 0.2586, "gold_yes": 0.5172}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 116

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Swahili · prompt_injections · yes

Solar Decide, same named hosted route; Swahili; slice prompt_injections. {"answered": 1997, "cases": 1997, "complete": true}

Source observations: {"n": 116, "acc": 0.6121, "en_acc": 0.7069, "auroc": 0.7908, "en_auroc": 0.5851, "yes": 0.1293, "en_yes": 0.2586, "gold_yes": 0.5172}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 116

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Swahili · prompt_injections · en_yes

Solar Decide, same named hosted route; Swahili; slice prompt_injections. {"answered": 1997, "cases": 1997, "complete": true}

Source observations: {"n": 116, "acc": 0.6121, "en_acc": 0.7069, "auroc": 0.7908, "en_auroc": 0.5851, "yes": 0.1293, "en_yes": 0.2586, "gold_yes": 0.5172}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 116

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Spanish · overall · acc

Solar Decide, same named hosted route; Spanish; slice overall. {"answered": 1959, "cases": 1959, "complete": true}

Source observations: {"n": 1959, "acc": 0.7379, "en_acc": 0.7968, "delta": -0.0589, "ci": [-0.0786, -0.0395], "significant": true, "auroc": 0.752, "en_auroc": 0.8001}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 1959

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Spanish · overall · auroc

Solar Decide, same named hosted route; Spanish; slice overall. {"answered": 1959, "cases": 1959, "complete": true}

Source observations: {"n": 1959, "acc": 0.7379, "en_acc": 0.7968, "delta": -0.0589, "ci": [-0.0786, -0.0395], "significant": true, "auroc": 0.752, "en_auroc": 0.8001}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 1959

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Spanish · overall · en_acc

Solar Decide, same named hosted route; Spanish; slice overall. {"answered": 1959, "cases": 1959, "complete": true}

Source observations: {"n": 1959, "acc": 0.7379, "en_acc": 0.7968, "delta": -0.0589, "ci": [-0.0786, -0.0395], "significant": true, "auroc": 0.752, "en_auroc": 0.8001}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 1959

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Spanish · overall · en_auroc

Solar Decide, same named hosted route; Spanish; slice overall. {"answered": 1959, "cases": 1959, "complete": true}

Source observations: {"n": 1959, "acc": 0.7379, "en_acc": 0.7968, "delta": -0.0589, "ci": [-0.0786, -0.0395], "significant": true, "auroc": 0.752, "en_auroc": 0.8001}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 1959

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Spanish · overall · delta

Solar Decide, same named hosted route; Spanish; slice overall. {"answered": 1959, "cases": 1959, "complete": true}

Source observations: {"n": 1959, "acc": 0.7379, "en_acc": 0.7968, "delta": -0.0589, "ci": [-0.0786, -0.0395], "significant": true, "auroc": 0.752, "en_auroc": 0.8001}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 1959

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Spanish · aegis2 · acc

Solar Decide, same named hosted route; Spanish; slice aegis2. {"answered": 1959, "cases": 1959, "complete": true}

Source observations: {"n": 499, "acc": 0.7535, "en_acc": 0.8076, "auroc": 0.8315, "en_auroc": 0.8441, "yes": 0.2946, "en_yes": 0.3647, "gold_yes": 0.501}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 499

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Spanish · aegis2 · auroc

Solar Decide, same named hosted route; Spanish; slice aegis2. {"answered": 1959, "cases": 1959, "complete": true}

Source observations: {"n": 499, "acc": 0.7535, "en_acc": 0.8076, "auroc": 0.8315, "en_auroc": 0.8441, "yes": 0.2946, "en_yes": 0.3647, "gold_yes": 0.501}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 499

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Spanish · aegis2 · en_acc

Solar Decide, same named hosted route; Spanish; slice aegis2. {"answered": 1959, "cases": 1959, "complete": true}

Source observations: {"n": 499, "acc": 0.7535, "en_acc": 0.8076, "auroc": 0.8315, "en_auroc": 0.8441, "yes": 0.2946, "en_yes": 0.3647, "gold_yes": 0.501}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 499

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Spanish · aegis2 · en_auroc

Solar Decide, same named hosted route; Spanish; slice aegis2. {"answered": 1959, "cases": 1959, "complete": true}

Source observations: {"n": 499, "acc": 0.7535, "en_acc": 0.8076, "auroc": 0.8315, "en_auroc": 0.8441, "yes": 0.2946, "en_yes": 0.3647, "gold_yes": 0.501}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 499

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Spanish · aegis2 · yes

Solar Decide, same named hosted route; Spanish; slice aegis2. {"answered": 1959, "cases": 1959, "complete": true}

Source observations: {"n": 499, "acc": 0.7535, "en_acc": 0.8076, "auroc": 0.8315, "en_auroc": 0.8441, "yes": 0.2946, "en_yes": 0.3647, "gold_yes": 0.501}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 499

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Spanish · aegis2 · en_yes

Solar Decide, same named hosted route; Spanish; slice aegis2. {"answered": 1959, "cases": 1959, "complete": true}

Source observations: {"n": 499, "acc": 0.7535, "en_acc": 0.8076, "auroc": 0.8315, "en_auroc": 0.8441, "yes": 0.2946, "en_yes": 0.3647, "gold_yes": 0.501}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 499

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Spanish · aegis2_response · acc

Solar Decide, same named hosted route; Spanish; slice aegis2_response. {"answered": 1959, "cases": 1959, "complete": true}

Source observations: {"n": 500, "acc": 0.764, "en_acc": 0.754, "auroc": 0.8119, "en_auroc": 0.8007, "yes": 0.328, "en_yes": 0.298, "gold_yes": 0.5}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 500

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Spanish · aegis2_response · auroc

Solar Decide, same named hosted route; Spanish; slice aegis2_response. {"answered": 1959, "cases": 1959, "complete": true}

Source observations: {"n": 500, "acc": 0.764, "en_acc": 0.754, "auroc": 0.8119, "en_auroc": 0.8007, "yes": 0.328, "en_yes": 0.298, "gold_yes": 0.5}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 500

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Spanish · aegis2_response · en_acc

Solar Decide, same named hosted route; Spanish; slice aegis2_response. {"answered": 1959, "cases": 1959, "complete": true}

Source observations: {"n": 500, "acc": 0.764, "en_acc": 0.754, "auroc": 0.8119, "en_auroc": 0.8007, "yes": 0.328, "en_yes": 0.298, "gold_yes": 0.5}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 500

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Spanish · aegis2_response · en_auroc

Solar Decide, same named hosted route; Spanish; slice aegis2_response. {"answered": 1959, "cases": 1959, "complete": true}

Source observations: {"n": 500, "acc": 0.764, "en_acc": 0.754, "auroc": 0.8119, "en_auroc": 0.8007, "yes": 0.328, "en_yes": 0.298, "gold_yes": 0.5}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 500

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Spanish · aegis2_response · yes

Solar Decide, same named hosted route; Spanish; slice aegis2_response. {"answered": 1959, "cases": 1959, "complete": true}

Source observations: {"n": 500, "acc": 0.764, "en_acc": 0.754, "auroc": 0.8119, "en_auroc": 0.8007, "yes": 0.328, "en_yes": 0.298, "gold_yes": 0.5}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 500

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Spanish · aegis2_response · en_yes

Solar Decide, same named hosted route; Spanish; slice aegis2_response. {"answered": 1959, "cases": 1959, "complete": true}

Source observations: {"n": 500, "acc": 0.764, "en_acc": 0.754, "auroc": 0.8119, "en_auroc": 0.8007, "yes": 0.328, "en_yes": 0.298, "gold_yes": 0.5}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 500

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Spanish · atbench500 · acc

Solar Decide, same named hosted route; Spanish; slice atbench500. {"answered": 1959, "cases": 1959, "complete": true}

Source observations: {"n": 451, "acc": 0.541, "en_acc": 0.7384, "auroc": 0.5852, "en_auroc": 0.8077, "yes": 0.7894, "en_yes": 0.5388, "gold_yes": 0.4723}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 451

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Spanish · atbench500 · auroc

Solar Decide, same named hosted route; Spanish; slice atbench500. {"answered": 1959, "cases": 1959, "complete": true}

Source observations: {"n": 451, "acc": 0.541, "en_acc": 0.7384, "auroc": 0.5852, "en_auroc": 0.8077, "yes": 0.7894, "en_yes": 0.5388, "gold_yes": 0.4723}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 451

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Spanish · atbench500 · en_acc

Solar Decide, same named hosted route; Spanish; slice atbench500. {"answered": 1959, "cases": 1959, "complete": true}

Source observations: {"n": 451, "acc": 0.541, "en_acc": 0.7384, "auroc": 0.5852, "en_auroc": 0.8077, "yes": 0.7894, "en_yes": 0.5388, "gold_yes": 0.4723}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 451

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Spanish · atbench500 · en_auroc

Solar Decide, same named hosted route; Spanish; slice atbench500. {"answered": 1959, "cases": 1959, "complete": true}

Source observations: {"n": 451, "acc": 0.541, "en_acc": 0.7384, "auroc": 0.5852, "en_auroc": 0.8077, "yes": 0.7894, "en_yes": 0.5388, "gold_yes": 0.4723}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 451

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Spanish · atbench500 · yes

Solar Decide, same named hosted route; Spanish; slice atbench500. {"answered": 1959, "cases": 1959, "complete": true}

Source observations: {"n": 451, "acc": 0.541, "en_acc": 0.7384, "auroc": 0.5852, "en_auroc": 0.8077, "yes": 0.7894, "en_yes": 0.5388, "gold_yes": 0.4723}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 451

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Spanish · atbench500 · en_yes

Solar Decide, same named hosted route; Spanish; slice atbench500. {"answered": 1959, "cases": 1959, "complete": true}

Source observations: {"n": 451, "acc": 0.541, "en_acc": 0.7384, "auroc": 0.5852, "en_auroc": 0.8077, "yes": 0.7894, "en_yes": 0.5388, "gold_yes": 0.4723}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 451

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Spanish · jailbreak_classification · acc

Solar Decide, same named hosted route; Spanish; slice jailbreak_classification. {"answered": 1959, "cases": 1959, "complete": true}

Source observations: {"n": 396, "acc": 0.9495, "en_acc": 0.9672, "auroc": 0.9276, "en_auroc": 0.9552, "yes": 0.3636, "en_yes": 0.3258, "gold_yes": 0.3485}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 396

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Spanish · jailbreak_classification · auroc

Solar Decide, same named hosted route; Spanish; slice jailbreak_classification. {"answered": 1959, "cases": 1959, "complete": true}

Source observations: {"n": 396, "acc": 0.9495, "en_acc": 0.9672, "auroc": 0.9276, "en_auroc": 0.9552, "yes": 0.3636, "en_yes": 0.3258, "gold_yes": 0.3485}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 396

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Spanish · jailbreak_classification · en_acc

Solar Decide, same named hosted route; Spanish; slice jailbreak_classification. {"answered": 1959, "cases": 1959, "complete": true}

Source observations: {"n": 396, "acc": 0.9495, "en_acc": 0.9672, "auroc": 0.9276, "en_auroc": 0.9552, "yes": 0.3636, "en_yes": 0.3258, "gold_yes": 0.3485}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 396

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Spanish · jailbreak_classification · en_auroc

Solar Decide, same named hosted route; Spanish; slice jailbreak_classification. {"answered": 1959, "cases": 1959, "complete": true}

Source observations: {"n": 396, "acc": 0.9495, "en_acc": 0.9672, "auroc": 0.9276, "en_auroc": 0.9552, "yes": 0.3636, "en_yes": 0.3258, "gold_yes": 0.3485}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 396

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Spanish · jailbreak_classification · yes

Solar Decide, same named hosted route; Spanish; slice jailbreak_classification. {"answered": 1959, "cases": 1959, "complete": true}

Source observations: {"n": 396, "acc": 0.9495, "en_acc": 0.9672, "auroc": 0.9276, "en_auroc": 0.9552, "yes": 0.3636, "en_yes": 0.3258, "gold_yes": 0.3485}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 396

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Spanish · jailbreak_classification · en_yes

Solar Decide, same named hosted route; Spanish; slice jailbreak_classification. {"answered": 1959, "cases": 1959, "complete": true}

Source observations: {"n": 396, "acc": 0.9495, "en_acc": 0.9672, "auroc": 0.9276, "en_auroc": 0.9552, "yes": 0.3636, "en_yes": 0.3258, "gold_yes": 0.3485}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 396

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Spanish · prompt_injections · acc

Solar Decide, same named hosted route; Spanish; slice prompt_injections. {"answered": 1959, "cases": 1959, "complete": true}

Source observations: {"n": 113, "acc": 0.6814, "en_acc": 0.7168, "auroc": 0.6037, "en_auroc": 0.5927, "yes": 0.2212, "en_yes": 0.2566, "gold_yes": 0.5044}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 113

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Spanish · prompt_injections · auroc

Solar Decide, same named hosted route; Spanish; slice prompt_injections. {"answered": 1959, "cases": 1959, "complete": true}

Source observations: {"n": 113, "acc": 0.6814, "en_acc": 0.7168, "auroc": 0.6037, "en_auroc": 0.5927, "yes": 0.2212, "en_yes": 0.2566, "gold_yes": 0.5044}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 113

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Spanish · prompt_injections · en_acc

Solar Decide, same named hosted route; Spanish; slice prompt_injections. {"answered": 1959, "cases": 1959, "complete": true}

Source observations: {"n": 113, "acc": 0.6814, "en_acc": 0.7168, "auroc": 0.6037, "en_auroc": 0.5927, "yes": 0.2212, "en_yes": 0.2566, "gold_yes": 0.5044}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 113

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Spanish · prompt_injections · en_auroc

Solar Decide, same named hosted route; Spanish; slice prompt_injections. {"answered": 1959, "cases": 1959, "complete": true}

Source observations: {"n": 113, "acc": 0.6814, "en_acc": 0.7168, "auroc": 0.6037, "en_auroc": 0.5927, "yes": 0.2212, "en_yes": 0.2566, "gold_yes": 0.5044}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 113

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Spanish · prompt_injections · yes

Solar Decide, same named hosted route; Spanish; slice prompt_injections. {"answered": 1959, "cases": 1959, "complete": true}

Source observations: {"n": 113, "acc": 0.6814, "en_acc": 0.7168, "auroc": 0.6037, "en_auroc": 0.5927, "yes": 0.2212, "en_yes": 0.2566, "gold_yes": 0.5044}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 113

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Spanish · prompt_injections · en_yes

Solar Decide, same named hosted route; Spanish; slice prompt_injections. {"answered": 1959, "cases": 1959, "complete": true}

Source observations: {"n": 113, "acc": 0.6814, "en_acc": 0.7168, "auroc": 0.6037, "en_auroc": 0.5927, "yes": 0.2212, "en_yes": 0.2566, "gold_yes": 0.5044}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 113

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · French · overall · acc

Solar Decide, same named hosted route; French; slice overall. {"answered": 1969, "cases": 1969, "complete": true}

Source observations: {"n": 1969, "acc": 0.7558, "en_acc": 0.7961, "delta": -0.0403, "ci": [-0.0586, -0.0218], "significant": true, "auroc": 0.7906, "en_auroc": 0.7995}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 1969

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · French · overall · auroc

Solar Decide, same named hosted route; French; slice overall. {"answered": 1969, "cases": 1969, "complete": true}

Source observations: {"n": 1969, "acc": 0.7558, "en_acc": 0.7961, "delta": -0.0403, "ci": [-0.0586, -0.0218], "significant": true, "auroc": 0.7906, "en_auroc": 0.7995}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 1969

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · French · overall · en_acc

Solar Decide, same named hosted route; French; slice overall. {"answered": 1969, "cases": 1969, "complete": true}

Source observations: {"n": 1969, "acc": 0.7558, "en_acc": 0.7961, "delta": -0.0403, "ci": [-0.0586, -0.0218], "significant": true, "auroc": 0.7906, "en_auroc": 0.7995}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 1969

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · French · overall · en_auroc

Solar Decide, same named hosted route; French; slice overall. {"answered": 1969, "cases": 1969, "complete": true}

Source observations: {"n": 1969, "acc": 0.7558, "en_acc": 0.7961, "delta": -0.0403, "ci": [-0.0586, -0.0218], "significant": true, "auroc": 0.7906, "en_auroc": 0.7995}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 1969

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · French · overall · delta

Solar Decide, same named hosted route; French; slice overall. {"answered": 1969, "cases": 1969, "complete": true}

Source observations: {"n": 1969, "acc": 0.7558, "en_acc": 0.7961, "delta": -0.0403, "ci": [-0.0586, -0.0218], "significant": true, "auroc": 0.7906, "en_auroc": 0.7995}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 1969

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · French · aegis2 · acc

Solar Decide, same named hosted route; French; slice aegis2. {"answered": 1969, "cases": 1969, "complete": true}

Source observations: {"n": 500, "acc": 0.752, "en_acc": 0.808, "auroc": 0.8317, "en_auroc": 0.8447, "yes": 0.284, "en_yes": 0.364, "gold_yes": 0.5}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 500

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · French · aegis2 · auroc

Solar Decide, same named hosted route; French; slice aegis2. {"answered": 1969, "cases": 1969, "complete": true}

Source observations: {"n": 500, "acc": 0.752, "en_acc": 0.808, "auroc": 0.8317, "en_auroc": 0.8447, "yes": 0.284, "en_yes": 0.364, "gold_yes": 0.5}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 500

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · French · aegis2 · en_acc

Solar Decide, same named hosted route; French; slice aegis2. {"answered": 1969, "cases": 1969, "complete": true}

Source observations: {"n": 500, "acc": 0.752, "en_acc": 0.808, "auroc": 0.8317, "en_auroc": 0.8447, "yes": 0.284, "en_yes": 0.364, "gold_yes": 0.5}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 500

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · French · aegis2 · en_auroc

Solar Decide, same named hosted route; French; slice aegis2. {"answered": 1969, "cases": 1969, "complete": true}

Source observations: {"n": 500, "acc": 0.752, "en_acc": 0.808, "auroc": 0.8317, "en_auroc": 0.8447, "yes": 0.284, "en_yes": 0.364, "gold_yes": 0.5}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 500

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · French · aegis2 · yes

Solar Decide, same named hosted route; French; slice aegis2. {"answered": 1969, "cases": 1969, "complete": true}

Source observations: {"n": 500, "acc": 0.752, "en_acc": 0.808, "auroc": 0.8317, "en_auroc": 0.8447, "yes": 0.284, "en_yes": 0.364, "gold_yes": 0.5}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 500

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · French · aegis2 · en_yes

Solar Decide, same named hosted route; French; slice aegis2. {"answered": 1969, "cases": 1969, "complete": true}

Source observations: {"n": 500, "acc": 0.752, "en_acc": 0.808, "auroc": 0.8317, "en_auroc": 0.8447, "yes": 0.284, "en_yes": 0.364, "gold_yes": 0.5}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 500

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · French · aegis2_response · acc

Solar Decide, same named hosted route; French; slice aegis2_response. {"answered": 1969, "cases": 1969, "complete": true}

Source observations: {"n": 500, "acc": 0.774, "en_acc": 0.754, "auroc": 0.7931, "en_auroc": 0.8007, "yes": 0.358, "en_yes": 0.298, "gold_yes": 0.5}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 500

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · French · aegis2_response · auroc

Solar Decide, same named hosted route; French; slice aegis2_response. {"answered": 1969, "cases": 1969, "complete": true}

Source observations: {"n": 500, "acc": 0.774, "en_acc": 0.754, "auroc": 0.7931, "en_auroc": 0.8007, "yes": 0.358, "en_yes": 0.298, "gold_yes": 0.5}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 500

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · French · aegis2_response · en_acc

Solar Decide, same named hosted route; French; slice aegis2_response. {"answered": 1969, "cases": 1969, "complete": true}

Source observations: {"n": 500, "acc": 0.774, "en_acc": 0.754, "auroc": 0.7931, "en_auroc": 0.8007, "yes": 0.358, "en_yes": 0.298, "gold_yes": 0.5}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 500

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · French · aegis2_response · en_auroc

Solar Decide, same named hosted route; French; slice aegis2_response. {"answered": 1969, "cases": 1969, "complete": true}

Source observations: {"n": 500, "acc": 0.774, "en_acc": 0.754, "auroc": 0.7931, "en_auroc": 0.8007, "yes": 0.358, "en_yes": 0.298, "gold_yes": 0.5}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 500

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · French · aegis2_response · yes

Solar Decide, same named hosted route; French; slice aegis2_response. {"answered": 1969, "cases": 1969, "complete": true}

Source observations: {"n": 500, "acc": 0.774, "en_acc": 0.754, "auroc": 0.7931, "en_auroc": 0.8007, "yes": 0.358, "en_yes": 0.298, "gold_yes": 0.5}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 500

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · French · aegis2_response · en_yes

Solar Decide, same named hosted route; French; slice aegis2_response. {"answered": 1969, "cases": 1969, "complete": true}

Source observations: {"n": 500, "acc": 0.774, "en_acc": 0.754, "auroc": 0.7931, "en_auroc": 0.8007, "yes": 0.358, "en_yes": 0.298, "gold_yes": 0.5}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 500

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · French · atbench500 · acc

Solar Decide, same named hosted route; French; slice atbench500. {"answered": 1969, "cases": 1969, "complete": true}

Source observations: {"n": 459, "acc": 0.695, "en_acc": 0.7386, "auroc": 0.8027, "en_auroc": 0.8035, "yes": 0.5011, "en_yes": 0.5577, "gold_yes": 0.5142}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 459

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · French · atbench500 · auroc

Solar Decide, same named hosted route; French; slice atbench500. {"answered": 1969, "cases": 1969, "complete": true}

Source observations: {"n": 459, "acc": 0.695, "en_acc": 0.7386, "auroc": 0.8027, "en_auroc": 0.8035, "yes": 0.5011, "en_yes": 0.5577, "gold_yes": 0.5142}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 459

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · French · atbench500 · en_acc

Solar Decide, same named hosted route; French; slice atbench500. {"answered": 1969, "cases": 1969, "complete": true}

Source observations: {"n": 459, "acc": 0.695, "en_acc": 0.7386, "auroc": 0.8027, "en_auroc": 0.8035, "yes": 0.5011, "en_yes": 0.5577, "gold_yes": 0.5142}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 459

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · French · atbench500 · en_auroc

Solar Decide, same named hosted route; French; slice atbench500. {"answered": 1969, "cases": 1969, "complete": true}

Source observations: {"n": 459, "acc": 0.695, "en_acc": 0.7386, "auroc": 0.8027, "en_auroc": 0.8035, "yes": 0.5011, "en_yes": 0.5577, "gold_yes": 0.5142}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 459

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · French · atbench500 · yes

Solar Decide, same named hosted route; French; slice atbench500. {"answered": 1969, "cases": 1969, "complete": true}

Source observations: {"n": 459, "acc": 0.695, "en_acc": 0.7386, "auroc": 0.8027, "en_auroc": 0.8035, "yes": 0.5011, "en_yes": 0.5577, "gold_yes": 0.5142}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 459

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · French · atbench500 · en_yes

Solar Decide, same named hosted route; French; slice atbench500. {"answered": 1969, "cases": 1969, "complete": true}

Source observations: {"n": 459, "acc": 0.695, "en_acc": 0.7386, "auroc": 0.8027, "en_auroc": 0.8035, "yes": 0.5011, "en_yes": 0.5577, "gold_yes": 0.5142}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 459

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · French · jailbreak_classification · acc

Solar Decide, same named hosted route; French; slice jailbreak_classification. {"answered": 1969, "cases": 1969, "complete": true}

Source observations: {"n": 395, "acc": 0.9494, "en_acc": 0.9671, "auroc": 0.9481, "en_auroc": 0.9548, "yes": 0.357, "en_yes": 0.3241, "gold_yes": 0.3468}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 395

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · French · jailbreak_classification · auroc

Solar Decide, same named hosted route; French; slice jailbreak_classification. {"answered": 1969, "cases": 1969, "complete": true}

Source observations: {"n": 395, "acc": 0.9494, "en_acc": 0.9671, "auroc": 0.9481, "en_auroc": 0.9548, "yes": 0.357, "en_yes": 0.3241, "gold_yes": 0.3468}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 395

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · French · jailbreak_classification · en_acc

Solar Decide, same named hosted route; French; slice jailbreak_classification. {"answered": 1969, "cases": 1969, "complete": true}

Source observations: {"n": 395, "acc": 0.9494, "en_acc": 0.9671, "auroc": 0.9481, "en_auroc": 0.9548, "yes": 0.357, "en_yes": 0.3241, "gold_yes": 0.3468}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 395

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · French · jailbreak_classification · en_auroc

Solar Decide, same named hosted route; French; slice jailbreak_classification. {"answered": 1969, "cases": 1969, "complete": true}

Source observations: {"n": 395, "acc": 0.9494, "en_acc": 0.9671, "auroc": 0.9481, "en_auroc": 0.9548, "yes": 0.357, "en_yes": 0.3241, "gold_yes": 0.3468}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 395

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · French · jailbreak_classification · yes

Solar Decide, same named hosted route; French; slice jailbreak_classification. {"answered": 1969, "cases": 1969, "complete": true}

Source observations: {"n": 395, "acc": 0.9494, "en_acc": 0.9671, "auroc": 0.9481, "en_auroc": 0.9548, "yes": 0.357, "en_yes": 0.3241, "gold_yes": 0.3468}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 395

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · French · jailbreak_classification · en_yes

Solar Decide, same named hosted route; French; slice jailbreak_classification. {"answered": 1969, "cases": 1969, "complete": true}

Source observations: {"n": 395, "acc": 0.9494, "en_acc": 0.9671, "auroc": 0.9481, "en_auroc": 0.9548, "yes": 0.357, "en_yes": 0.3241, "gold_yes": 0.3468}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 395

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · French · prompt_injections · acc

Solar Decide, same named hosted route; French; slice prompt_injections. {"answered": 1969, "cases": 1969, "complete": true}

Source observations: {"n": 115, "acc": 0.6087, "en_acc": 0.713, "auroc": 0.5772, "en_auroc": 0.5938, "yes": 0.2783, "en_yes": 0.2609, "gold_yes": 0.513}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 115

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · French · prompt_injections · auroc

Solar Decide, same named hosted route; French; slice prompt_injections. {"answered": 1969, "cases": 1969, "complete": true}

Source observations: {"n": 115, "acc": 0.6087, "en_acc": 0.713, "auroc": 0.5772, "en_auroc": 0.5938, "yes": 0.2783, "en_yes": 0.2609, "gold_yes": 0.513}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 115

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · French · prompt_injections · en_acc

Solar Decide, same named hosted route; French; slice prompt_injections. {"answered": 1969, "cases": 1969, "complete": true}

Source observations: {"n": 115, "acc": 0.6087, "en_acc": 0.713, "auroc": 0.5772, "en_auroc": 0.5938, "yes": 0.2783, "en_yes": 0.2609, "gold_yes": 0.513}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 115

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · French · prompt_injections · en_auroc

Solar Decide, same named hosted route; French; slice prompt_injections. {"answered": 1969, "cases": 1969, "complete": true}

Source observations: {"n": 115, "acc": 0.6087, "en_acc": 0.713, "auroc": 0.5772, "en_auroc": 0.5938, "yes": 0.2783, "en_yes": 0.2609, "gold_yes": 0.513}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 115

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · French · prompt_injections · yes

Solar Decide, same named hosted route; French; slice prompt_injections. {"answered": 1969, "cases": 1969, "complete": true}

Source observations: {"n": 115, "acc": 0.6087, "en_acc": 0.713, "auroc": 0.5772, "en_auroc": 0.5938, "yes": 0.2783, "en_yes": 0.2609, "gold_yes": 0.513}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 115

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · French · prompt_injections · en_yes

Solar Decide, same named hosted route; French; slice prompt_injections. {"answered": 1969, "cases": 1969, "complete": true}

Source observations: {"n": 115, "acc": 0.6087, "en_acc": 0.713, "auroc": 0.5772, "en_auroc": 0.5938, "yes": 0.2783, "en_yes": 0.2609, "gold_yes": 0.513}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 115

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Russian · overall · acc

Solar Decide, same named hosted route; Russian; slice overall. {"answered": 2005, "cases": 2005, "complete": true}

Source observations: {"n": 2005, "acc": 0.7612, "en_acc": 0.7949, "delta": -0.0337, "ci": [-0.0543, -0.0129], "significant": true, "auroc": 0.7891, "en_auroc": 0.7971}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 2005

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Russian · overall · auroc

Solar Decide, same named hosted route; Russian; slice overall. {"answered": 2005, "cases": 2005, "complete": true}

Source observations: {"n": 2005, "acc": 0.7612, "en_acc": 0.7949, "delta": -0.0337, "ci": [-0.0543, -0.0129], "significant": true, "auroc": 0.7891, "en_auroc": 0.7971}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 2005

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Russian · overall · en_acc

Solar Decide, same named hosted route; Russian; slice overall. {"answered": 2005, "cases": 2005, "complete": true}

Source observations: {"n": 2005, "acc": 0.7612, "en_acc": 0.7949, "delta": -0.0337, "ci": [-0.0543, -0.0129], "significant": true, "auroc": 0.7891, "en_auroc": 0.7971}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 2005

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Russian · overall · en_auroc

Solar Decide, same named hosted route; Russian; slice overall. {"answered": 2005, "cases": 2005, "complete": true}

Source observations: {"n": 2005, "acc": 0.7612, "en_acc": 0.7949, "delta": -0.0337, "ci": [-0.0543, -0.0129], "significant": true, "auroc": 0.7891, "en_auroc": 0.7971}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 2005

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Russian · overall · delta

Solar Decide, same named hosted route; Russian; slice overall. {"answered": 2005, "cases": 2005, "complete": true}

Source observations: {"n": 2005, "acc": 0.7612, "en_acc": 0.7949, "delta": -0.0337, "ci": [-0.0543, -0.0129], "significant": true, "auroc": 0.7891, "en_auroc": 0.7971}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 2005

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Russian · aegis2 · acc

Solar Decide, same named hosted route; Russian; slice aegis2. {"answered": 2005, "cases": 2005, "complete": true}

Source observations: {"n": 500, "acc": 0.792, "en_acc": 0.808, "auroc": 0.8377, "en_auroc": 0.8447, "yes": 0.344, "en_yes": 0.364, "gold_yes": 0.5}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 500

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Russian · aegis2 · auroc

Solar Decide, same named hosted route; Russian; slice aegis2. {"answered": 2005, "cases": 2005, "complete": true}

Source observations: {"n": 500, "acc": 0.792, "en_acc": 0.808, "auroc": 0.8377, "en_auroc": 0.8447, "yes": 0.344, "en_yes": 0.364, "gold_yes": 0.5}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 500

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Russian · aegis2 · en_acc

Solar Decide, same named hosted route; Russian; slice aegis2. {"answered": 2005, "cases": 2005, "complete": true}

Source observations: {"n": 500, "acc": 0.792, "en_acc": 0.808, "auroc": 0.8377, "en_auroc": 0.8447, "yes": 0.344, "en_yes": 0.364, "gold_yes": 0.5}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 500

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Russian · aegis2 · en_auroc

Solar Decide, same named hosted route; Russian; slice aegis2. {"answered": 2005, "cases": 2005, "complete": true}

Source observations: {"n": 500, "acc": 0.792, "en_acc": 0.808, "auroc": 0.8377, "en_auroc": 0.8447, "yes": 0.344, "en_yes": 0.364, "gold_yes": 0.5}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 500

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Russian · aegis2 · yes

Solar Decide, same named hosted route; Russian; slice aegis2. {"answered": 2005, "cases": 2005, "complete": true}

Source observations: {"n": 500, "acc": 0.792, "en_acc": 0.808, "auroc": 0.8377, "en_auroc": 0.8447, "yes": 0.344, "en_yes": 0.364, "gold_yes": 0.5}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 500

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Russian · aegis2 · en_yes

Solar Decide, same named hosted route; Russian; slice aegis2. {"answered": 2005, "cases": 2005, "complete": true}

Source observations: {"n": 500, "acc": 0.792, "en_acc": 0.808, "auroc": 0.8377, "en_auroc": 0.8447, "yes": 0.344, "en_yes": 0.364, "gold_yes": 0.5}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 500

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Russian · aegis2_response · acc

Solar Decide, same named hosted route; Russian; slice aegis2_response. {"answered": 2005, "cases": 2005, "complete": true}

Source observations: {"n": 500, "acc": 0.772, "en_acc": 0.754, "auroc": 0.8156, "en_auroc": 0.8007, "yes": 0.364, "en_yes": 0.298, "gold_yes": 0.5}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 500

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Russian · aegis2_response · auroc

Solar Decide, same named hosted route; Russian; slice aegis2_response. {"answered": 2005, "cases": 2005, "complete": true}

Source observations: {"n": 500, "acc": 0.772, "en_acc": 0.754, "auroc": 0.8156, "en_auroc": 0.8007, "yes": 0.364, "en_yes": 0.298, "gold_yes": 0.5}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 500

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Russian · aegis2_response · en_acc

Solar Decide, same named hosted route; Russian; slice aegis2_response. {"answered": 2005, "cases": 2005, "complete": true}

Source observations: {"n": 500, "acc": 0.772, "en_acc": 0.754, "auroc": 0.8156, "en_auroc": 0.8007, "yes": 0.364, "en_yes": 0.298, "gold_yes": 0.5}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 500

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Russian · aegis2_response · en_auroc

Solar Decide, same named hosted route; Russian; slice aegis2_response. {"answered": 2005, "cases": 2005, "complete": true}

Source observations: {"n": 500, "acc": 0.772, "en_acc": 0.754, "auroc": 0.8156, "en_auroc": 0.8007, "yes": 0.364, "en_yes": 0.298, "gold_yes": 0.5}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 500

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Russian · aegis2_response · yes

Solar Decide, same named hosted route; Russian; slice aegis2_response. {"answered": 2005, "cases": 2005, "complete": true}

Source observations: {"n": 500, "acc": 0.772, "en_acc": 0.754, "auroc": 0.8156, "en_auroc": 0.8007, "yes": 0.364, "en_yes": 0.298, "gold_yes": 0.5}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 500

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Russian · aegis2_response · en_yes

Solar Decide, same named hosted route; Russian; slice aegis2_response. {"answered": 2005, "cases": 2005, "complete": true}

Source observations: {"n": 500, "acc": 0.772, "en_acc": 0.754, "auroc": 0.8156, "en_auroc": 0.8007, "yes": 0.364, "en_yes": 0.298, "gold_yes": 0.5}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 500

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Russian · atbench500 · acc

Solar Decide, same named hosted route; Russian; slice atbench500. {"answered": 2005, "cases": 2005, "complete": true}

Source observations: {"n": 490, "acc": 0.6163, "en_acc": 0.7408, "auroc": 0.6728, "en_auroc": 0.8057, "yes": 0.6959, "en_yes": 0.551, "gold_yes": 0.5041}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 490

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Russian · atbench500 · auroc

Solar Decide, same named hosted route; Russian; slice atbench500. {"answered": 2005, "cases": 2005, "complete": true}

Source observations: {"n": 490, "acc": 0.6163, "en_acc": 0.7408, "auroc": 0.6728, "en_auroc": 0.8057, "yes": 0.6959, "en_yes": 0.551, "gold_yes": 0.5041}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 490

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Russian · atbench500 · en_acc

Solar Decide, same named hosted route; Russian; slice atbench500. {"answered": 2005, "cases": 2005, "complete": true}

Source observations: {"n": 490, "acc": 0.6163, "en_acc": 0.7408, "auroc": 0.6728, "en_auroc": 0.8057, "yes": 0.6959, "en_yes": 0.551, "gold_yes": 0.5041}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 490

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Russian · atbench500 · en_auroc

Solar Decide, same named hosted route; Russian; slice atbench500. {"answered": 2005, "cases": 2005, "complete": true}

Source observations: {"n": 490, "acc": 0.6163, "en_acc": 0.7408, "auroc": 0.6728, "en_auroc": 0.8057, "yes": 0.6959, "en_yes": 0.551, "gold_yes": 0.5041}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 490

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Russian · atbench500 · yes

Solar Decide, same named hosted route; Russian; slice atbench500. {"answered": 2005, "cases": 2005, "complete": true}

Source observations: {"n": 490, "acc": 0.6163, "en_acc": 0.7408, "auroc": 0.6728, "en_auroc": 0.8057, "yes": 0.6959, "en_yes": 0.551, "gold_yes": 0.5041}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 490

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Russian · atbench500 · en_yes

Solar Decide, same named hosted route; Russian; slice atbench500. {"answered": 2005, "cases": 2005, "complete": true}

Source observations: {"n": 490, "acc": 0.6163, "en_acc": 0.7408, "auroc": 0.6728, "en_auroc": 0.8057, "yes": 0.6959, "en_yes": 0.551, "gold_yes": 0.5041}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 490

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Russian · jailbreak_classification · acc

Solar Decide, same named hosted route; Russian; slice jailbreak_classification. {"answered": 2005, "cases": 2005, "complete": true}

Source observations: {"n": 399, "acc": 0.9273, "en_acc": 0.9649, "auroc": 0.9512, "en_auroc": 0.9491, "yes": 0.3935, "en_yes": 0.3258, "gold_yes": 0.3509}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 399

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Russian · jailbreak_classification · auroc

Solar Decide, same named hosted route; Russian; slice jailbreak_classification. {"answered": 2005, "cases": 2005, "complete": true}

Source observations: {"n": 399, "acc": 0.9273, "en_acc": 0.9649, "auroc": 0.9512, "en_auroc": 0.9491, "yes": 0.3935, "en_yes": 0.3258, "gold_yes": 0.3509}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 399

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Russian · jailbreak_classification · en_acc

Solar Decide, same named hosted route; Russian; slice jailbreak_classification. {"answered": 2005, "cases": 2005, "complete": true}

Source observations: {"n": 399, "acc": 0.9273, "en_acc": 0.9649, "auroc": 0.9512, "en_auroc": 0.9491, "yes": 0.3935, "en_yes": 0.3258, "gold_yes": 0.3509}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 399

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Russian · jailbreak_classification · en_auroc

Solar Decide, same named hosted route; Russian; slice jailbreak_classification. {"answered": 2005, "cases": 2005, "complete": true}

Source observations: {"n": 399, "acc": 0.9273, "en_acc": 0.9649, "auroc": 0.9512, "en_auroc": 0.9491, "yes": 0.3935, "en_yes": 0.3258, "gold_yes": 0.3509}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 399

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Russian · jailbreak_classification · yes

Solar Decide, same named hosted route; Russian; slice jailbreak_classification. {"answered": 2005, "cases": 2005, "complete": true}

Source observations: {"n": 399, "acc": 0.9273, "en_acc": 0.9649, "auroc": 0.9512, "en_auroc": 0.9491, "yes": 0.3935, "en_yes": 0.3258, "gold_yes": 0.3509}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 399

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Russian · jailbreak_classification · en_yes

Solar Decide, same named hosted route; Russian; slice jailbreak_classification. {"answered": 2005, "cases": 2005, "complete": true}

Source observations: {"n": 399, "acc": 0.9273, "en_acc": 0.9649, "auroc": 0.9512, "en_auroc": 0.9491, "yes": 0.3935, "en_yes": 0.3258, "gold_yes": 0.3509}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 399

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Russian · prompt_injections · acc

Solar Decide, same named hosted route; Russian; slice prompt_injections. {"answered": 2005, "cases": 2005, "complete": true}

Source observations: {"n": 116, "acc": 0.6983, "en_acc": 0.7069, "auroc": 0.6685, "en_auroc": 0.5851, "yes": 0.2845, "en_yes": 0.2586, "gold_yes": 0.5172}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 116

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Russian · prompt_injections · auroc

Solar Decide, same named hosted route; Russian; slice prompt_injections. {"answered": 2005, "cases": 2005, "complete": true}

Source observations: {"n": 116, "acc": 0.6983, "en_acc": 0.7069, "auroc": 0.6685, "en_auroc": 0.5851, "yes": 0.2845, "en_yes": 0.2586, "gold_yes": 0.5172}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 116

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Russian · prompt_injections · en_acc

Solar Decide, same named hosted route; Russian; slice prompt_injections. {"answered": 2005, "cases": 2005, "complete": true}

Source observations: {"n": 116, "acc": 0.6983, "en_acc": 0.7069, "auroc": 0.6685, "en_auroc": 0.5851, "yes": 0.2845, "en_yes": 0.2586, "gold_yes": 0.5172}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 116

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Russian · prompt_injections · en_auroc

Solar Decide, same named hosted route; Russian; slice prompt_injections. {"answered": 2005, "cases": 2005, "complete": true}

Source observations: {"n": 116, "acc": 0.6983, "en_acc": 0.7069, "auroc": 0.6685, "en_auroc": 0.5851, "yes": 0.2845, "en_yes": 0.2586, "gold_yes": 0.5172}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 116

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Russian · prompt_injections · yes

Solar Decide, same named hosted route; Russian; slice prompt_injections. {"answered": 2005, "cases": 2005, "complete": true}

Source observations: {"n": 116, "acc": 0.6983, "en_acc": 0.7069, "auroc": 0.6685, "en_auroc": 0.5851, "yes": 0.2845, "en_yes": 0.2586, "gold_yes": 0.5172}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 116

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Russian · prompt_injections · en_yes

Solar Decide, same named hosted route; Russian; slice prompt_injections. {"answered": 2005, "cases": 2005, "complete": true}

Source observations: {"n": 116, "acc": 0.6983, "en_acc": 0.7069, "auroc": 0.6685, "en_auroc": 0.5851, "yes": 0.2845, "en_yes": 0.2586, "gold_yes": 0.5172}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 116

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained