Examenos

Exact named hosted API observation; All-case accuracy.

Named source route upstage/solar-decide-flash; ModelTested runner v1; tested_on=openrouter. Backend weight revision and API inference precision unpublished.

Publisher generated 2026-11-01; board added this route 2026-10-11. Execution dates, token usage and hashes remain in its per-run receipts. No private prompt/response archive imported.

56 authored situations, including 19 hard cases, each with two typed questions; 14 public and 42 hidden cases. Source aggregate averages per-case decision accuracy. Confidence score is 100×(1−mean multiclass Brier/2), not ECE or a calibration probability. Small single-run benchmark; current HF access/size does not inherit this evaluator’s stale closed/API-only metadata. No tariff inferred from rounded cost 0.0. Rows added 1 November 2026 use revision 668a52ad3b932ca910fd985f96dcdf0a85d2dae1, where the earlier 11 systems are unchanged. Rows added 2 November 2026 use revision 5c6c19bb3b7332ccd28c8c13bd32cdc69cedc8f0.

Checked / reported: 2026-11-01 · Samples: 56

Snapshot: 2026-10-18 · Revision: 081e8f37130a4a9077b8cffe62827d4c3b9e3b23; 668a52ad3b932ca910fd985f96dcdf0a85d2dae1 for Drex Max, Jev 1.14 and d1.1; 5c6c19bb3b7332ccd28c8c13bd32cdc69cedc8f0 for Mammoth, Elephant, Joey, Span-02 and both Solar Decide routes · Published aggregate observations; result reuse terms not separately specified

Exact named hosted API observation; Hard-case accuracy.

Named source route upstage/solar-decide-flash; ModelTested runner v1; tested_on=openrouter. Backend weight revision and API inference precision unpublished.

Publisher generated 2026-11-01; board added this route 2026-10-11. Execution dates, token usage and hashes remain in its per-run receipts. No private prompt/response archive imported.

56 authored situations, including 19 hard cases, each with two typed questions; 14 public and 42 hidden cases. Source aggregate averages per-case decision accuracy. Confidence score is 100×(1−mean multiclass Brier/2), not ECE or a calibration probability. Small single-run benchmark; current HF access/size does not inherit this evaluator’s stale closed/API-only metadata. No tariff inferred from rounded cost 0.0. Rows added 1 November 2026 use revision 668a52ad3b932ca910fd985f96dcdf0a85d2dae1, where the earlier 11 systems are unchanged. Rows added 2 November 2026 use revision 5c6c19bb3b7332ccd28c8c13bd32cdc69cedc8f0.

Checked / reported: 2026-11-01 · Samples: 56

Snapshot: 2026-10-18 · Revision: 081e8f37130a4a9077b8cffe62827d4c3b9e3b23; 668a52ad3b932ca910fd985f96dcdf0a85d2dae1 for Drex Max, Jev 1.14 and d1.1; 5c6c19bb3b7332ccd28c8c13bd32cdc69cedc8f0 for Mammoth, Elephant, Joey, Span-02 and both Solar Decide routes · Published aggregate observations; result reuse terms not separately specified

Exact named hosted API observation; Public-case accuracy.

Named source route upstage/solar-decide-flash; ModelTested runner v1; tested_on=openrouter. Backend weight revision and API inference precision unpublished.

Publisher generated 2026-11-01; board added this route 2026-10-11. Execution dates, token usage and hashes remain in its per-run receipts. No private prompt/response archive imported.

56 authored situations, including 19 hard cases, each with two typed questions; 14 public and 42 hidden cases. Source aggregate averages per-case decision accuracy. Confidence score is 100×(1−mean multiclass Brier/2), not ECE or a calibration probability. Small single-run benchmark; current HF access/size does not inherit this evaluator’s stale closed/API-only metadata. No tariff inferred from rounded cost 0.0. Rows added 1 November 2026 use revision 668a52ad3b932ca910fd985f96dcdf0a85d2dae1, where the earlier 11 systems are unchanged. Rows added 2 November 2026 use revision 5c6c19bb3b7332ccd28c8c13bd32cdc69cedc8f0.

Checked / reported: 2026-11-01 · Samples: 56

Snapshot: 2026-10-18 · Revision: 081e8f37130a4a9077b8cffe62827d4c3b9e3b23; 668a52ad3b932ca910fd985f96dcdf0a85d2dae1 for Drex Max, Jev 1.14 and d1.1; 5c6c19bb3b7332ccd28c8c13bd32cdc69cedc8f0 for Mammoth, Elephant, Joey, Span-02 and both Solar Decide routes · Published aggregate observations; result reuse terms not separately specified

Exact named hosted API observation; Hidden-case accuracy.

Named source route upstage/solar-decide-flash; ModelTested runner v1; tested_on=openrouter. Backend weight revision and API inference precision unpublished.

Publisher generated 2026-11-01; board added this route 2026-10-11. Execution dates, token usage and hashes remain in its per-run receipts. No private prompt/response archive imported.

56 authored situations, including 19 hard cases, each with two typed questions; 14 public and 42 hidden cases. Source aggregate averages per-case decision accuracy. Confidence score is 100×(1−mean multiclass Brier/2), not ECE or a calibration probability. Small single-run benchmark; current HF access/size does not inherit this evaluator’s stale closed/API-only metadata. No tariff inferred from rounded cost 0.0. Rows added 1 November 2026 use revision 668a52ad3b932ca910fd985f96dcdf0a85d2dae1, where the earlier 11 systems are unchanged. Rows added 2 November 2026 use revision 5c6c19bb3b7332ccd28c8c13bd32cdc69cedc8f0.

Checked / reported: 2026-11-01 · Samples: 56

Snapshot: 2026-10-18 · Revision: 081e8f37130a4a9077b8cffe62827d4c3b9e3b23; 668a52ad3b932ca910fd985f96dcdf0a85d2dae1 for Drex Max, Jev 1.14 and d1.1; 5c6c19bb3b7332ccd28c8c13bd32cdc69cedc8f0 for Mammoth, Elephant, Joey, Span-02 and both Solar Decide routes · Published aggregate observations; result reuse terms not separately specified

Exact named hosted API observation; Brier-based confidence score.

Named source route upstage/solar-decide-flash; ModelTested runner v1; tested_on=openrouter. Backend weight revision and API inference precision unpublished.

Publisher generated 2026-11-01; board added this route 2026-10-11. Execution dates, token usage and hashes remain in its per-run receipts. No private prompt/response archive imported.

56 authored situations, including 19 hard cases, each with two typed questions; 14 public and 42 hidden cases. Source aggregate averages per-case decision accuracy. Confidence score is 100×(1−mean multiclass Brier/2), not ECE or a calibration probability. Small single-run benchmark; current HF access/size does not inherit this evaluator’s stale closed/API-only metadata. No tariff inferred from rounded cost 0.0. Rows added 1 November 2026 use revision 668a52ad3b932ca910fd985f96dcdf0a85d2dae1, where the earlier 11 systems are unchanged. Rows added 2 November 2026 use revision 5c6c19bb3b7332ccd28c8c13bd32cdc69cedc8f0.

Checked / reported: 2026-11-01 · Samples: 56

Snapshot: 2026-10-18 · Revision: 081e8f37130a4a9077b8cffe62827d4c3b9e3b23; 668a52ad3b932ca910fd985f96dcdf0a85d2dae1 for Drex Max, Jev 1.14 and d1.1; 5c6c19bb3b7332ccd28c8c13bd32cdc69cedc8f0 for Mammoth, Elephant, Joey, Span-02 and both Solar Decide routes · Published aggregate observations; result reuse terms not separately specified

JevBench Headline: aegis2 · accuracy

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "aegis2", "p50 ms": "640", "p95 ms": "6175", "p99 ms": "6397", "mean input tokens": "505", "total input tokens": "252,268", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:43-0700 · Samples: 500 · Source 95% CI: 69.8–77.4

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Headline: aegis2_response · accuracy

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "aegis2_response", "p50 ms": "585", "p95 ms": "728", "p99 ms": "1444", "mean input tokens": "607", "total input tokens": "303,429", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:43-0700 · Samples: 500 · Source 95% CI: 68–76

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Headline: atbench500 · accuracy

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "atbench500", "p50 ms": "638", "p95 ms": "891", "p99 ms": "1134", "mean input tokens": "2223", "total input tokens": "1,111,555", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:43-0700 · Samples: 500 · Source 95% CI: 45.6–54.400000000000006

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Headline: helpsteer2_helpfulness · mae_expected_level

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "helpsteer2_helpfulness", "p50 ms": "661", "p95 ms": "5182", "p99 ms": "5759", "mean input tokens": "934", "total input tokens": "467,105", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:43-0700 · Samples: 500 · Source 95% CI: 1.02–1.175

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Headline: jailbreak_classification · accuracy

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "jailbreak_classification", "p50 ms": "574", "p95 ms": "712", "p99 ms": "1415", "mean input tokens": "713", "total input tokens": "285,150", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:43-0700 · Samples: 400 · Source 95% CI: 95.3–98.5

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Headline: mcpbench_tool_selection · tool_selection_accuracy

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "mcpbench_tool_selection", "p50 ms": "564", "p95 ms": "640", "p99 ms": "640", "mean input tokens": "915", "total input tokens": "23,778", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:43-0700 · Samples: 26

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Headline: mcpmark_server_routing · server_routing_accuracy

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "mcpmark_server_routing", "p50 ms": "560", "p95 ms": "691", "p99 ms": "2157", "mean input tokens": "470", "total input tokens": "159,016", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:43-0700 · Samples: 338 · Source 95% CI: 33.4–47.599999999999994

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Headline: mcpmark_tool_selection · tool_selection_accuracy

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "mcpmark_tool_selection", "p50 ms": "574", "p95 ms": "2077", "p99 ms": "2077", "mean input tokens": "647", "total input tokens": "11,642", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:43-0700 · Samples: 18

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Headline: medmcqa · accuracy

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "medmcqa", "p50 ms": "559", "p95 ms": "744", "p99 ms": "1398", "mean input tokens": "399", "total input tokens": "399,370", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:43-0700 · Samples: 1000 · Source 95% CI: 59.8–67.2

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Headline: medqa_usmle · accuracy

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "medqa_usmle", "p50 ms": "572", "p95 ms": "747", "p99 ms": "1651", "mean input tokens": "554", "total input tokens": "553,650", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:43-0700 · Samples: 1000 · Source 95% CI: 68.60000000000001–75.8

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Headline: mmlu_pro · accuracy

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "mmlu_pro", "p50 ms": "569", "p95 ms": "716", "p99 ms": "1138", "mean input tokens": "545", "total input tokens": "544,710", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:43-0700 · Samples: 1000 · Source 95% CI: 43.5–51.300000000000004

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Headline: nfcorpus_rank_choice · ndcg@10

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "nfcorpus_rank_choice", "p50 ms": "1073", "p95 ms": "1516", "p99 ms": "2043", "mean input tokens": "2503", "total input tokens": "720,835", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:43-0700 · Samples: 298 · Source 95% CI: 27–33.7

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Headline: nfcorpus_rank_graded · ndcg@10

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "nfcorpus_rank_graded", "p50 ms": "576", "p95 ms": "960", "p99 ms": "1457", "mean input tokens": "570", "total input tokens": "2,807,330", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:43-0700 · Samples: 4924 · Source 95% CI: 33–40.1

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Headline: nfcorpus_rank_noul · ndcg@10

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "nfcorpus_rank_noul", "p50 ms": "2069", "p95 ms": "3121", "p99 ms": "3644", "mean input tokens": "48457", "total input tokens": "14,440,038", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:43-0700 · Samples: 298 · Source 95% CI: 29.5–36.6

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Headline: prompt_injections · accuracy

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "prompt_injections", "p50 ms": "627", "p95 ms": "817", "p99 ms": "918", "mean input tokens": "497", "total input tokens": "57,662", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:43-0700 · Samples: 116 · Source 95% CI: 71.6–86.2

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Headline: pubmedqa · accuracy

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "pubmedqa", "p50 ms": "570", "p95 ms": "694", "p99 ms": "1042", "mean input tokens": "711", "total input tokens": "355,398", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:43-0700 · Samples: 500 · Source 95% CI: 48–56.8

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Headline: rewardbench2 · accuracy

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "rewardbench2", "p50 ms": "673", "p95 ms": "1086", "p99 ms": "1435", "mean input tokens": "1957", "total input tokens": "2,348,908", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:43-0700 · Samples: 1200 · Source 95% CI: 60.099999999999994–66.4

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Headline: scienceqa_text · accuracy

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "scienceqa_text", "p50 ms": "556", "p95 ms": "649", "p99 ms": "983", "mean input tokens": "402", "total input tokens": "402,396", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:43-0700 · Samples: 1000 · Source 95% CI: 82.89999999999999–88.1

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Headline: scifact_rank_choice · ndcg@10

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "scifact_rank_choice", "p50 ms": "1145", "p95 ms": "1631", "p99 ms": "1951", "mean input tokens": "2897", "total input tokens": "869,148", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:43-0700 · Samples: 300 · Source 95% CI: 58.699999999999996–68.7

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Headline: scifact_rank_noul · ndcg@10

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "scifact_rank_noul", "p50 ms": "2173", "p95 ms": "2894", "p99 ms": "3232", "mean input tokens": "61557", "total input tokens": "18,467,080", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:43-0700 · Samples: 300 · Source 95% CI: 61.7–70.39999999999999

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Headline: sst5 · mae_expected_level

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "sst5", "p50 ms": "586", "p95 ms": "1008", "p99 ms": "1214", "mean input tokens": "390", "total input tokens": "194,814", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:43-0700 · Samples: 500 · Source 95% CI: 0.655–0.791

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: mcpbench_tool_selection · accuracy

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "mcpbench_tool_selection", "p50 ms": "564", "p95 ms": "640", "p99 ms": "640", "mean input tokens": "915", "total input tokens": "23,778", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:43-0700 · Samples: 26

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: mcpbench_tool_selection · macro-F1

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "mcpbench_tool_selection", "p50 ms": "564", "p95 ms": "640", "p99 ms": "640", "mean input tokens": "915", "total input tokens": "23,778", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:43-0700 · Samples: 26

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: mcpbench_tool_selection · Brier

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "mcpbench_tool_selection", "p50 ms": "564", "p95 ms": "640", "p99 ms": "640", "mean input tokens": "915", "total input tokens": "23,778", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:43-0700 · Samples: 26

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: mcpbench_tool_selection · NLL

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "mcpbench_tool_selection", "p50 ms": "564", "p95 ms": "640", "p99 ms": "640", "mean input tokens": "915", "total input tokens": "23,778", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:43-0700 · Samples: 26

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: mcpbench_tool_selection · ECE (10 bins)

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "mcpbench_tool_selection", "p50 ms": "564", "p95 ms": "640", "p99 ms": "640", "mean input tokens": "915", "total input tokens": "23,778", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:43-0700 · Samples: 26

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: mcpmark_server_routing · accuracy

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "mcpmark_server_routing", "p50 ms": "560", "p95 ms": "691", "p99 ms": "2157", "mean input tokens": "470", "total input tokens": "159,016", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:43-0700 · Samples: 338

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: mcpmark_server_routing · macro-F1

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "mcpmark_server_routing", "p50 ms": "560", "p95 ms": "691", "p99 ms": "2157", "mean input tokens": "470", "total input tokens": "159,016", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:43-0700 · Samples: 338

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: mcpmark_server_routing · Brier

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "mcpmark_server_routing", "p50 ms": "560", "p95 ms": "691", "p99 ms": "2157", "mean input tokens": "470", "total input tokens": "159,016", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:43-0700 · Samples: 338

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: mcpmark_server_routing · NLL

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "mcpmark_server_routing", "p50 ms": "560", "p95 ms": "691", "p99 ms": "2157", "mean input tokens": "470", "total input tokens": "159,016", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:43-0700 · Samples: 338

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: mcpmark_server_routing · ECE (10 bins)

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "mcpmark_server_routing", "p50 ms": "560", "p95 ms": "691", "p99 ms": "2157", "mean input tokens": "470", "total input tokens": "159,016", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:43-0700 · Samples: 338

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: mcpmark_tool_selection · accuracy

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "mcpmark_tool_selection", "p50 ms": "574", "p95 ms": "2077", "p99 ms": "2077", "mean input tokens": "647", "total input tokens": "11,642", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:43-0700 · Samples: 18

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: mcpmark_tool_selection · macro-F1

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "mcpmark_tool_selection", "p50 ms": "574", "p95 ms": "2077", "p99 ms": "2077", "mean input tokens": "647", "total input tokens": "11,642", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:43-0700 · Samples: 18

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: mcpmark_tool_selection · Brier

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "mcpmark_tool_selection", "p50 ms": "574", "p95 ms": "2077", "p99 ms": "2077", "mean input tokens": "647", "total input tokens": "11,642", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:43-0700 · Samples: 18

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: mcpmark_tool_selection · NLL

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "mcpmark_tool_selection", "p50 ms": "574", "p95 ms": "2077", "p99 ms": "2077", "mean input tokens": "647", "total input tokens": "11,642", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:43-0700 · Samples: 18

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: mcpmark_tool_selection · ECE (10 bins)

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "mcpmark_tool_selection", "p50 ms": "574", "p95 ms": "2077", "p99 ms": "2077", "mean input tokens": "647", "total input tokens": "11,642", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:43-0700 · Samples: 18

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: medmcqa · accuracy

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "medmcqa", "p50 ms": "559", "p95 ms": "744", "p99 ms": "1398", "mean input tokens": "399", "total input tokens": "399,370", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:43-0700 · Samples: 1000

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: medmcqa · macro-F1

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "medmcqa", "p50 ms": "559", "p95 ms": "744", "p99 ms": "1398", "mean input tokens": "399", "total input tokens": "399,370", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:43-0700 · Samples: 1000

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: medmcqa · Brier

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "medmcqa", "p50 ms": "559", "p95 ms": "744", "p99 ms": "1398", "mean input tokens": "399", "total input tokens": "399,370", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:43-0700 · Samples: 1000

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: medmcqa · NLL

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "medmcqa", "p50 ms": "559", "p95 ms": "744", "p99 ms": "1398", "mean input tokens": "399", "total input tokens": "399,370", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:43-0700 · Samples: 1000

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: medmcqa · ECE (10 bins)

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "medmcqa", "p50 ms": "559", "p95 ms": "744", "p99 ms": "1398", "mean input tokens": "399", "total input tokens": "399,370", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:43-0700 · Samples: 1000

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: medqa_usmle · accuracy

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "medqa_usmle", "p50 ms": "572", "p95 ms": "747", "p99 ms": "1651", "mean input tokens": "554", "total input tokens": "553,650", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:43-0700 · Samples: 1000

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: medqa_usmle · macro-F1

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "medqa_usmle", "p50 ms": "572", "p95 ms": "747", "p99 ms": "1651", "mean input tokens": "554", "total input tokens": "553,650", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:43-0700 · Samples: 1000

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: medqa_usmle · Brier

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "medqa_usmle", "p50 ms": "572", "p95 ms": "747", "p99 ms": "1651", "mean input tokens": "554", "total input tokens": "553,650", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:43-0700 · Samples: 1000

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: medqa_usmle · NLL

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "medqa_usmle", "p50 ms": "572", "p95 ms": "747", "p99 ms": "1651", "mean input tokens": "554", "total input tokens": "553,650", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:43-0700 · Samples: 1000

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: medqa_usmle · ECE (10 bins)

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "medqa_usmle", "p50 ms": "572", "p95 ms": "747", "p99 ms": "1651", "mean input tokens": "554", "total input tokens": "553,650", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:43-0700 · Samples: 1000

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: mmlu_pro · accuracy

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "mmlu_pro", "p50 ms": "569", "p95 ms": "716", "p99 ms": "1138", "mean input tokens": "545", "total input tokens": "544,710", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:43-0700 · Samples: 1000

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: mmlu_pro · macro-F1

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "mmlu_pro", "p50 ms": "569", "p95 ms": "716", "p99 ms": "1138", "mean input tokens": "545", "total input tokens": "544,710", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:43-0700 · Samples: 1000

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: mmlu_pro · Brier

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "mmlu_pro", "p50 ms": "569", "p95 ms": "716", "p99 ms": "1138", "mean input tokens": "545", "total input tokens": "544,710", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:43-0700 · Samples: 1000

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: mmlu_pro · NLL

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "mmlu_pro", "p50 ms": "569", "p95 ms": "716", "p99 ms": "1138", "mean input tokens": "545", "total input tokens": "544,710", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:43-0700 · Samples: 1000

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: mmlu_pro · ECE (10 bins)

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "mmlu_pro", "p50 ms": "569", "p95 ms": "716", "p99 ms": "1138", "mean input tokens": "545", "total input tokens": "544,710", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:43-0700 · Samples: 1000

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: pubmedqa · accuracy

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "pubmedqa", "p50 ms": "570", "p95 ms": "694", "p99 ms": "1042", "mean input tokens": "711", "total input tokens": "355,398", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:43-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: pubmedqa · macro-F1

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "pubmedqa", "p50 ms": "570", "p95 ms": "694", "p99 ms": "1042", "mean input tokens": "711", "total input tokens": "355,398", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:43-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: pubmedqa · Brier

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "pubmedqa", "p50 ms": "570", "p95 ms": "694", "p99 ms": "1042", "mean input tokens": "711", "total input tokens": "355,398", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:43-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: pubmedqa · NLL

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "pubmedqa", "p50 ms": "570", "p95 ms": "694", "p99 ms": "1042", "mean input tokens": "711", "total input tokens": "355,398", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:43-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: pubmedqa · ECE (10 bins)

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "pubmedqa", "p50 ms": "570", "p95 ms": "694", "p99 ms": "1042", "mean input tokens": "711", "total input tokens": "355,398", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:43-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: rewardbench2 · accuracy

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "rewardbench2", "p50 ms": "673", "p95 ms": "1086", "p99 ms": "1435", "mean input tokens": "1957", "total input tokens": "2,348,908", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:43-0700 · Samples: 1200

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: rewardbench2 · macro-F1

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "rewardbench2", "p50 ms": "673", "p95 ms": "1086", "p99 ms": "1435", "mean input tokens": "1957", "total input tokens": "2,348,908", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:43-0700 · Samples: 1200

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: rewardbench2 · Brier

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "rewardbench2", "p50 ms": "673", "p95 ms": "1086", "p99 ms": "1435", "mean input tokens": "1957", "total input tokens": "2,348,908", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:43-0700 · Samples: 1200

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: rewardbench2 · NLL

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "rewardbench2", "p50 ms": "673", "p95 ms": "1086", "p99 ms": "1435", "mean input tokens": "1957", "total input tokens": "2,348,908", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:43-0700 · Samples: 1200

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: rewardbench2 · ECE (10 bins)

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "rewardbench2", "p50 ms": "673", "p95 ms": "1086", "p99 ms": "1435", "mean input tokens": "1957", "total input tokens": "2,348,908", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:43-0700 · Samples: 1200

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: scienceqa_text · accuracy

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "scienceqa_text", "p50 ms": "556", "p95 ms": "649", "p99 ms": "983", "mean input tokens": "402", "total input tokens": "402,396", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:43-0700 · Samples: 1000

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: scienceqa_text · macro-F1

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "scienceqa_text", "p50 ms": "556", "p95 ms": "649", "p99 ms": "983", "mean input tokens": "402", "total input tokens": "402,396", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:43-0700 · Samples: 1000

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: scienceqa_text · Brier

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "scienceqa_text", "p50 ms": "556", "p95 ms": "649", "p99 ms": "983", "mean input tokens": "402", "total input tokens": "402,396", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:43-0700 · Samples: 1000

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: scienceqa_text · NLL

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "scienceqa_text", "p50 ms": "556", "p95 ms": "649", "p99 ms": "983", "mean input tokens": "402", "total input tokens": "402,396", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:43-0700 · Samples: 1000

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: scienceqa_text · ECE (10 bins)

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "scienceqa_text", "p50 ms": "556", "p95 ms": "649", "p99 ms": "983", "mean input tokens": "402", "total input tokens": "402,396", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:43-0700 · Samples: 1000

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: aegis2 · acc @0.5

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "aegis2", "p50 ms": "640", "p95 ms": "6175", "p99 ms": "6397", "mean input tokens": "505", "total input tokens": "252,268", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:43-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: aegis2 · acc if flipped

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "aegis2", "p50 ms": "640", "p95 ms": "6175", "p99 ms": "6397", "mean input tokens": "505", "total input tokens": "252,268", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:43-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: aegis2 · positive recall

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "aegis2", "p50 ms": "640", "p95 ms": "6175", "p99 ms": "6397", "mean input tokens": "505", "total input tokens": "252,268", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:43-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: aegis2 · FPR

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "aegis2", "p50 ms": "640", "p95 ms": "6175", "p99 ms": "6397", "mean input tokens": "505", "total input tokens": "252,268", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:43-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: aegis2 · AUROC

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "aegis2", "p50 ms": "640", "p95 ms": "6175", "p99 ms": "6397", "mean input tokens": "505", "total input tokens": "252,268", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:43-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: aegis2 · Brier

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "aegis2", "p50 ms": "640", "p95 ms": "6175", "p99 ms": "6397", "mean input tokens": "505", "total input tokens": "252,268", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:43-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: aegis2 · ECE

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "aegis2", "p50 ms": "640", "p95 ms": "6175", "p99 ms": "6397", "mean input tokens": "505", "total input tokens": "252,268", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:43-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: aegis2_response · acc @0.5

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "aegis2_response", "p50 ms": "585", "p95 ms": "728", "p99 ms": "1444", "mean input tokens": "607", "total input tokens": "303,429", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:43-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: aegis2_response · acc if flipped

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "aegis2_response", "p50 ms": "585", "p95 ms": "728", "p99 ms": "1444", "mean input tokens": "607", "total input tokens": "303,429", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:43-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: aegis2_response · positive recall

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "aegis2_response", "p50 ms": "585", "p95 ms": "728", "p99 ms": "1444", "mean input tokens": "607", "total input tokens": "303,429", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:43-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: aegis2_response · FPR

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "aegis2_response", "p50 ms": "585", "p95 ms": "728", "p99 ms": "1444", "mean input tokens": "607", "total input tokens": "303,429", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:43-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: aegis2_response · AUROC

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "aegis2_response", "p50 ms": "585", "p95 ms": "728", "p99 ms": "1444", "mean input tokens": "607", "total input tokens": "303,429", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:43-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: aegis2_response · Brier

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "aegis2_response", "p50 ms": "585", "p95 ms": "728", "p99 ms": "1444", "mean input tokens": "607", "total input tokens": "303,429", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:43-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: aegis2_response · ECE

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "aegis2_response", "p50 ms": "585", "p95 ms": "728", "p99 ms": "1444", "mean input tokens": "607", "total input tokens": "303,429", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:43-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: atbench500 · acc @0.5

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "atbench500", "p50 ms": "638", "p95 ms": "891", "p99 ms": "1134", "mean input tokens": "2223", "total input tokens": "1,111,555", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:43-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: atbench500 · acc if flipped

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "atbench500", "p50 ms": "638", "p95 ms": "891", "p99 ms": "1134", "mean input tokens": "2223", "total input tokens": "1,111,555", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:43-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: atbench500 · positive recall

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "atbench500", "p50 ms": "638", "p95 ms": "891", "p99 ms": "1134", "mean input tokens": "2223", "total input tokens": "1,111,555", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:43-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: atbench500 · FPR

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "atbench500", "p50 ms": "638", "p95 ms": "891", "p99 ms": "1134", "mean input tokens": "2223", "total input tokens": "1,111,555", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:43-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: atbench500 · AUROC

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "atbench500", "p50 ms": "638", "p95 ms": "891", "p99 ms": "1134", "mean input tokens": "2223", "total input tokens": "1,111,555", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:43-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: atbench500 · Brier

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "atbench500", "p50 ms": "638", "p95 ms": "891", "p99 ms": "1134", "mean input tokens": "2223", "total input tokens": "1,111,555", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:43-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: atbench500 · ECE

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "atbench500", "p50 ms": "638", "p95 ms": "891", "p99 ms": "1134", "mean input tokens": "2223", "total input tokens": "1,111,555", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:43-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: jailbreak_classification · acc @0.5

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "jailbreak_classification", "p50 ms": "574", "p95 ms": "712", "p99 ms": "1415", "mean input tokens": "713", "total input tokens": "285,150", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:43-0700 · Samples: 400

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: jailbreak_classification · acc if flipped

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "jailbreak_classification", "p50 ms": "574", "p95 ms": "712", "p99 ms": "1415", "mean input tokens": "713", "total input tokens": "285,150", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:43-0700 · Samples: 400

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: jailbreak_classification · positive recall

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "jailbreak_classification", "p50 ms": "574", "p95 ms": "712", "p99 ms": "1415", "mean input tokens": "713", "total input tokens": "285,150", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:43-0700 · Samples: 400

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: jailbreak_classification · FPR

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "jailbreak_classification", "p50 ms": "574", "p95 ms": "712", "p99 ms": "1415", "mean input tokens": "713", "total input tokens": "285,150", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:43-0700 · Samples: 400

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: jailbreak_classification · AUROC

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "jailbreak_classification", "p50 ms": "574", "p95 ms": "712", "p99 ms": "1415", "mean input tokens": "713", "total input tokens": "285,150", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:43-0700 · Samples: 400

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: jailbreak_classification · Brier

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "jailbreak_classification", "p50 ms": "574", "p95 ms": "712", "p99 ms": "1415", "mean input tokens": "713", "total input tokens": "285,150", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:43-0700 · Samples: 400

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: jailbreak_classification · ECE

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "jailbreak_classification", "p50 ms": "574", "p95 ms": "712", "p99 ms": "1415", "mean input tokens": "713", "total input tokens": "285,150", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:43-0700 · Samples: 400

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: prompt_injections · acc @0.5

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "prompt_injections", "p50 ms": "627", "p95 ms": "817", "p99 ms": "918", "mean input tokens": "497", "total input tokens": "57,662", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:43-0700 · Samples: 116

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: prompt_injections · acc if flipped

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "prompt_injections", "p50 ms": "627", "p95 ms": "817", "p99 ms": "918", "mean input tokens": "497", "total input tokens": "57,662", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:43-0700 · Samples: 116

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: prompt_injections · positive recall

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "prompt_injections", "p50 ms": "627", "p95 ms": "817", "p99 ms": "918", "mean input tokens": "497", "total input tokens": "57,662", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:43-0700 · Samples: 116

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: prompt_injections · FPR

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "prompt_injections", "p50 ms": "627", "p95 ms": "817", "p99 ms": "918", "mean input tokens": "497", "total input tokens": "57,662", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:43-0700 · Samples: 116

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: prompt_injections · AUROC

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "prompt_injections", "p50 ms": "627", "p95 ms": "817", "p99 ms": "918", "mean input tokens": "497", "total input tokens": "57,662", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:43-0700 · Samples: 116

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: prompt_injections · Brier

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "prompt_injections", "p50 ms": "627", "p95 ms": "817", "p99 ms": "918", "mean input tokens": "497", "total input tokens": "57,662", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:43-0700 · Samples: 116

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: prompt_injections · ECE

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "prompt_injections", "p50 ms": "627", "p95 ms": "817", "p99 ms": "918", "mean input tokens": "497", "total input tokens": "57,662", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:43-0700 · Samples: 116

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ordinal slices: helpsteer2_helpfulness · exact (rounded)

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "helpsteer2_helpfulness", "p50 ms": "661", "p95 ms": "5182", "p99 ms": "5759", "mean input tokens": "934", "total input tokens": "467,105", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:43-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ordinal slices: helpsteer2_helpfulness · within 1 level

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "helpsteer2_helpfulness", "p50 ms": "661", "p95 ms": "5182", "p99 ms": "5759", "mean input tokens": "934", "total input tokens": "467,105", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:43-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ordinal slices: helpsteer2_helpfulness · MAE (expected level)

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "helpsteer2_helpfulness", "p50 ms": "661", "p95 ms": "5182", "p99 ms": "5759", "mean input tokens": "934", "total input tokens": "467,105", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:43-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ordinal slices: helpsteer2_helpfulness · RPS

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "helpsteer2_helpfulness", "p50 ms": "661", "p95 ms": "5182", "p99 ms": "5759", "mean input tokens": "934", "total input tokens": "467,105", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:43-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ordinal slices: helpsteer2_helpfulness · Spearman

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "helpsteer2_helpfulness", "p50 ms": "661", "p95 ms": "5182", "p99 ms": "5759", "mean input tokens": "934", "total input tokens": "467,105", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:43-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ordinal slices: nfcorpus_rank_graded · exact (rounded)

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "nfcorpus_rank_graded", "p50 ms": "576", "p95 ms": "960", "p99 ms": "1457", "mean input tokens": "570", "total input tokens": "2,807,330", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:43-0700 · Samples: 4924

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ordinal slices: nfcorpus_rank_graded · within 1 level

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "nfcorpus_rank_graded", "p50 ms": "576", "p95 ms": "960", "p99 ms": "1457", "mean input tokens": "570", "total input tokens": "2,807,330", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:43-0700 · Samples: 4924

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ordinal slices: nfcorpus_rank_graded · MAE (expected level)

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "nfcorpus_rank_graded", "p50 ms": "576", "p95 ms": "960", "p99 ms": "1457", "mean input tokens": "570", "total input tokens": "2,807,330", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:43-0700 · Samples: 4924

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ordinal slices: nfcorpus_rank_graded · RPS

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "nfcorpus_rank_graded", "p50 ms": "576", "p95 ms": "960", "p99 ms": "1457", "mean input tokens": "570", "total input tokens": "2,807,330", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:43-0700 · Samples: 4924

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ordinal slices: nfcorpus_rank_graded · Spearman

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "nfcorpus_rank_graded", "p50 ms": "576", "p95 ms": "960", "p99 ms": "1457", "mean input tokens": "570", "total input tokens": "2,807,330", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:43-0700 · Samples: 4924

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ordinal slices: sst5 · exact (rounded)

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "sst5", "p50 ms": "586", "p95 ms": "1008", "p99 ms": "1214", "mean input tokens": "390", "total input tokens": "194,814", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:43-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ordinal slices: sst5 · within 1 level

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "sst5", "p50 ms": "586", "p95 ms": "1008", "p99 ms": "1214", "mean input tokens": "390", "total input tokens": "194,814", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:43-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ordinal slices: sst5 · MAE (expected level)

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "sst5", "p50 ms": "586", "p95 ms": "1008", "p99 ms": "1214", "mean input tokens": "390", "total input tokens": "194,814", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:43-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ordinal slices: sst5 · RPS

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "sst5", "p50 ms": "586", "p95 ms": "1008", "p99 ms": "1214", "mean input tokens": "390", "total input tokens": "194,814", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:43-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ordinal slices: sst5 · Spearman

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "sst5", "p50 ms": "586", "p95 ms": "1008", "p99 ms": "1214", "mean input tokens": "390", "total input tokens": "194,814", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:43-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ranking slices: nfcorpus_rank_choice · nDCG@10

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "nfcorpus_rank_choice", "p50 ms": "1073", "p95 ms": "1516", "p99 ms": "2043", "mean input tokens": "2503", "total input tokens": "720,835", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:43-0700 · Samples: 288

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ranking slices: nfcorpus_rank_choice · MRR@10

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "nfcorpus_rank_choice", "p50 ms": "1073", "p95 ms": "1516", "p99 ms": "2043", "mean input tokens": "2503", "total input tokens": "720,835", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:43-0700 · Samples: 288

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ranking slices: nfcorpus_rank_choice · Recall@10

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "nfcorpus_rank_choice", "p50 ms": "1073", "p95 ms": "1516", "p99 ms": "2043", "mean input tokens": "2503", "total input tokens": "720,835", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:43-0700 · Samples: 288

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ranking slices: nfcorpus_rank_graded · nDCG@10

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "nfcorpus_rank_graded", "p50 ms": "576", "p95 ms": "960", "p99 ms": "1457", "mean input tokens": "570", "total input tokens": "2,807,330", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:43-0700 · Samples: 298

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ranking slices: nfcorpus_rank_graded · MRR@10

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "nfcorpus_rank_graded", "p50 ms": "576", "p95 ms": "960", "p99 ms": "1457", "mean input tokens": "570", "total input tokens": "2,807,330", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:43-0700 · Samples: 298

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ranking slices: nfcorpus_rank_graded · Recall@10

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "nfcorpus_rank_graded", "p50 ms": "576", "p95 ms": "960", "p99 ms": "1457", "mean input tokens": "570", "total input tokens": "2,807,330", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:43-0700 · Samples: 298

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ranking slices: nfcorpus_rank_noul · nDCG@10

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "nfcorpus_rank_noul", "p50 ms": "2069", "p95 ms": "3121", "p99 ms": "3644", "mean input tokens": "48457", "total input tokens": "14,440,038", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:43-0700 · Samples: 298

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ranking slices: nfcorpus_rank_noul · MRR@10

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "nfcorpus_rank_noul", "p50 ms": "2069", "p95 ms": "3121", "p99 ms": "3644", "mean input tokens": "48457", "total input tokens": "14,440,038", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:43-0700 · Samples: 298

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ranking slices: nfcorpus_rank_noul · Recall@10

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "nfcorpus_rank_noul", "p50 ms": "2069", "p95 ms": "3121", "p99 ms": "3644", "mean input tokens": "48457", "total input tokens": "14,440,038", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:43-0700 · Samples: 298

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ranking slices: scifact_rank_choice · nDCG@10

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "scifact_rank_choice", "p50 ms": "1145", "p95 ms": "1631", "p99 ms": "1951", "mean input tokens": "2897", "total input tokens": "869,148", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:43-0700 · Samples: 300

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ranking slices: scifact_rank_choice · MRR@10

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "scifact_rank_choice", "p50 ms": "1145", "p95 ms": "1631", "p99 ms": "1951", "mean input tokens": "2897", "total input tokens": "869,148", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:43-0700 · Samples: 300

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ranking slices: scifact_rank_choice · Recall@10

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "scifact_rank_choice", "p50 ms": "1145", "p95 ms": "1631", "p99 ms": "1951", "mean input tokens": "2897", "total input tokens": "869,148", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:43-0700 · Samples: 300

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ranking slices: scifact_rank_noul · nDCG@10

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "scifact_rank_noul", "p50 ms": "2173", "p95 ms": "2894", "p99 ms": "3232", "mean input tokens": "61557", "total input tokens": "18,467,080", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:43-0700 · Samples: 300

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ranking slices: scifact_rank_noul · MRR@10

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "scifact_rank_noul", "p50 ms": "2173", "p95 ms": "2894", "p99 ms": "3232", "mean input tokens": "61557", "total input tokens": "18,467,080", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:43-0700 · Samples: 300

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ranking slices: scifact_rank_noul · Recall@10

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "scifact_rank_noul", "p50 ms": "2173", "p95 ms": "2894", "p99 ms": "3232", "mean input tokens": "61557", "total input tokens": "18,467,080", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:43-0700 · Samples: 300

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench By subset: rewardbench2 · accuracy

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "rewardbench2", "p50 ms": "673", "p95 ms": "1086", "p99 ms": "1435", "mean input tokens": "1957", "total input tokens": "2,348,908", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:43-0700 · Samples: 240

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench By subset: rewardbench2 · accuracy

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "rewardbench2", "p50 ms": "673", "p95 ms": "1086", "p99 ms": "1435", "mean input tokens": "1957", "total input tokens": "2,348,908", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:43-0700 · Samples: 240

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench By subset: rewardbench2 · accuracy

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "rewardbench2", "p50 ms": "673", "p95 ms": "1086", "p99 ms": "1435", "mean input tokens": "1957", "total input tokens": "2,348,908", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:43-0700 · Samples: 240

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench By subset: rewardbench2 · accuracy

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "rewardbench2", "p50 ms": "673", "p95 ms": "1086", "p99 ms": "1435", "mean input tokens": "1957", "total input tokens": "2,348,908", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:43-0700 · Samples: 240

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench By subset: rewardbench2 · accuracy

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "rewardbench2", "p50 ms": "673", "p95 ms": "1086", "p99 ms": "1435", "mean input tokens": "1957", "total input tokens": "2,348,908", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:43-0700 · Samples: 240

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: mcpbench_tool_selection · original order

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "mcpbench_tool_selection", "p50 ms": "564", "p95 ms": "640", "p99 ms": "640", "mean input tokens": "915", "total input tokens": "23,778", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:43-0700 · Samples: 4

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: mcpbench_tool_selection · permuted order

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "mcpbench_tool_selection", "p50 ms": "564", "p95 ms": "640", "p99 ms": "640", "mean input tokens": "915", "total input tokens": "23,778", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:43-0700 · Samples: 4

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: mcpbench_tool_selection · both correct

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "mcpbench_tool_selection", "p50 ms": "564", "p95 ms": "640", "p99 ms": "640", "mean input tokens": "915", "total input tokens": "23,778", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:43-0700 · Samples: 4

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: mcpbench_tool_selection · flipped by reordering

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "mcpbench_tool_selection", "p50 ms": "564", "p95 ms": "640", "p99 ms": "640", "mean input tokens": "915", "total input tokens": "23,778", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:43-0700 · Samples: 4

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: mcpmark_server_routing · original order

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "mcpmark_server_routing", "p50 ms": "560", "p95 ms": "691", "p99 ms": "2157", "mean input tokens": "470", "total input tokens": "159,016", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:43-0700 · Samples: 169

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: mcpmark_server_routing · permuted order

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "mcpmark_server_routing", "p50 ms": "560", "p95 ms": "691", "p99 ms": "2157", "mean input tokens": "470", "total input tokens": "159,016", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:43-0700 · Samples: 169

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: mcpmark_server_routing · both correct

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "mcpmark_server_routing", "p50 ms": "560", "p95 ms": "691", "p99 ms": "2157", "mean input tokens": "470", "total input tokens": "159,016", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:43-0700 · Samples: 169

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: mcpmark_server_routing · flipped by reordering

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "mcpmark_server_routing", "p50 ms": "560", "p95 ms": "691", "p99 ms": "2157", "mean input tokens": "470", "total input tokens": "159,016", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:43-0700 · Samples: 169

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: mcpmark_tool_selection · original order

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "mcpmark_tool_selection", "p50 ms": "574", "p95 ms": "2077", "p99 ms": "2077", "mean input tokens": "647", "total input tokens": "11,642", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:43-0700 · Samples: 3

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: mcpmark_tool_selection · permuted order

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "mcpmark_tool_selection", "p50 ms": "574", "p95 ms": "2077", "p99 ms": "2077", "mean input tokens": "647", "total input tokens": "11,642", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:43-0700 · Samples: 3

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: mcpmark_tool_selection · both correct

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "mcpmark_tool_selection", "p50 ms": "574", "p95 ms": "2077", "p99 ms": "2077", "mean input tokens": "647", "total input tokens": "11,642", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:43-0700 · Samples: 3

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: mcpmark_tool_selection · flipped by reordering

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "mcpmark_tool_selection", "p50 ms": "574", "p95 ms": "2077", "p99 ms": "2077", "mean input tokens": "647", "total input tokens": "11,642", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:43-0700 · Samples: 3

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: medmcqa · original order

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "medmcqa", "p50 ms": "559", "p95 ms": "744", "p99 ms": "1398", "mean input tokens": "399", "total input tokens": "399,370", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:43-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: medmcqa · permuted order

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "medmcqa", "p50 ms": "559", "p95 ms": "744", "p99 ms": "1398", "mean input tokens": "399", "total input tokens": "399,370", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:43-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: medmcqa · both correct

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "medmcqa", "p50 ms": "559", "p95 ms": "744", "p99 ms": "1398", "mean input tokens": "399", "total input tokens": "399,370", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:43-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: medmcqa · flipped by reordering

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "medmcqa", "p50 ms": "559", "p95 ms": "744", "p99 ms": "1398", "mean input tokens": "399", "total input tokens": "399,370", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:43-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: medqa_usmle · original order

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "medqa_usmle", "p50 ms": "572", "p95 ms": "747", "p99 ms": "1651", "mean input tokens": "554", "total input tokens": "553,650", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:43-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: medqa_usmle · permuted order

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "medqa_usmle", "p50 ms": "572", "p95 ms": "747", "p99 ms": "1651", "mean input tokens": "554", "total input tokens": "553,650", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:43-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: medqa_usmle · both correct

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "medqa_usmle", "p50 ms": "572", "p95 ms": "747", "p99 ms": "1651", "mean input tokens": "554", "total input tokens": "553,650", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:43-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: medqa_usmle · flipped by reordering

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "medqa_usmle", "p50 ms": "572", "p95 ms": "747", "p99 ms": "1651", "mean input tokens": "554", "total input tokens": "553,650", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:43-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: mmlu_pro · original order

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "mmlu_pro", "p50 ms": "569", "p95 ms": "716", "p99 ms": "1138", "mean input tokens": "545", "total input tokens": "544,710", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:43-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: mmlu_pro · permuted order

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "mmlu_pro", "p50 ms": "569", "p95 ms": "716", "p99 ms": "1138", "mean input tokens": "545", "total input tokens": "544,710", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:43-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: mmlu_pro · both correct

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "mmlu_pro", "p50 ms": "569", "p95 ms": "716", "p99 ms": "1138", "mean input tokens": "545", "total input tokens": "544,710", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:43-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: mmlu_pro · flipped by reordering

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "mmlu_pro", "p50 ms": "569", "p95 ms": "716", "p99 ms": "1138", "mean input tokens": "545", "total input tokens": "544,710", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:43-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: rewardbench2 · original order

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "rewardbench2", "p50 ms": "673", "p95 ms": "1086", "p99 ms": "1435", "mean input tokens": "1957", "total input tokens": "2,348,908", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:43-0700 · Samples: 594

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: rewardbench2 · permuted order

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "rewardbench2", "p50 ms": "673", "p95 ms": "1086", "p99 ms": "1435", "mean input tokens": "1957", "total input tokens": "2,348,908", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:43-0700 · Samples: 594

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: rewardbench2 · both correct

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "rewardbench2", "p50 ms": "673", "p95 ms": "1086", "p99 ms": "1435", "mean input tokens": "1957", "total input tokens": "2,348,908", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:43-0700 · Samples: 594

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: rewardbench2 · flipped by reordering

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "rewardbench2", "p50 ms": "673", "p95 ms": "1086", "p99 ms": "1435", "mean input tokens": "1957", "total input tokens": "2,348,908", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:43-0700 · Samples: 594

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: scienceqa_text · original order

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "scienceqa_text", "p50 ms": "556", "p95 ms": "649", "p99 ms": "983", "mean input tokens": "402", "total input tokens": "402,396", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:43-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: scienceqa_text · permuted order

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "scienceqa_text", "p50 ms": "556", "p95 ms": "649", "p99 ms": "983", "mean input tokens": "402", "total input tokens": "402,396", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:43-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: scienceqa_text · both correct

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "scienceqa_text", "p50 ms": "556", "p95 ms": "649", "p99 ms": "983", "mean input tokens": "402", "total input tokens": "402,396", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:43-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: scienceqa_text · flipped by reordering

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published row of the native report. Runtime diagnostics {"slice": "scienceqa_text", "p50 ms": "556", "p95 ms": "649", "p99 ms": "983", "mean input tokens": "402", "total input tokens": "402,396", "retries": "0"}.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-11-02 · Evaluated: 2026-10-15T23:39:43-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Sep26 · Index

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published category aggregate. Zeroed by the source: ["BANKING77", "MCP-Bench routing"].

Publisher normalized per-dataset scores, averaged within six categories and then categories. Complete6/6; not Benchmark Heaven JevBench or an Examenos recomputation.

Checked / reported: 2026-11-02 · Samples: Not published

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench Sep26 · Science

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published category aggregate. Zeroed by the source: ["BANKING77", "MCP-Bench routing"].

Publisher normalized per-dataset scores, averaged within six categories and then categories. Complete6/6; not Benchmark Heaven JevBench or an Examenos recomputation.

Checked / reported: 2026-11-02 · Samples: Not published

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench Sep26 · Tool calling

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published category aggregate. Zeroed by the source: ["BANKING77", "MCP-Bench routing"].

Publisher normalized per-dataset scores, averaged within six categories and then categories. Complete6/6; not Benchmark Heaven JevBench or an Examenos recomputation.

Checked / reported: 2026-11-02 · Samples: Not published

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench Sep26 · Business

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published category aggregate. Zeroed by the source: ["BANKING77", "MCP-Bench routing"].

Publisher normalized per-dataset scores, averaged within six categories and then categories. Complete6/6; not Benchmark Heaven JevBench or an Examenos recomputation.

Checked / reported: 2026-11-02 · Samples: Not published

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench Sep26 · Selection

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published category aggregate. Zeroed by the source: ["BANKING77", "MCP-Bench routing"].

Publisher normalized per-dataset scores, averaged within six categories and then categories. Complete6/6; not Benchmark Heaven JevBench or an Examenos recomputation.

Checked / reported: 2026-11-02 · Samples: Not published

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench Sep26 · Ranking

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published category aggregate. Zeroed by the source: ["BANKING77", "MCP-Bench routing"].

Publisher normalized per-dataset scores, averaged within six categories and then categories. Complete6/6; not Benchmark Heaven JevBench or an Examenos recomputation.

Checked / reported: 2026-11-02 · Samples: Not published

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench Sep26 · Judging

Scored 2026-10-15T23:39:43-0700. Model requested `or-upstage-solar-decide-flash`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Published category aggregate. Zeroed by the source: ["BANKING77", "MCP-Bench routing"].

Publisher normalized per-dataset scores, averaged within six categories and then categories. Complete6/6; not Benchmark Heaven JevBench or an Examenos recomputation.

Checked / reported: 2026-11-02 · Samples: Not published

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · English baseline · overall · acc

Solar Decide Flash, same named hosted route; English baseline; slice overall. {}

Source observations: {"n": 2016, "acc": 0.7438, "auroc": 0.846}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 2016

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · English baseline · overall · auroc

Solar Decide Flash, same named hosted route; English baseline; slice overall. {}

Source observations: {"n": 2016, "acc": 0.7438, "auroc": 0.846}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 2016

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · English baseline · aegis2 · acc

Solar Decide Flash, same named hosted route; English baseline; slice aegis2. {}

Source observations: {"n": 500, "acc": 0.736, "auroc": 0.8451, "yes": 0.3, "gold_yes": 0.5}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 500

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · English baseline · aegis2 · auroc

Solar Decide Flash, same named hosted route; English baseline; slice aegis2. {}

Source observations: {"n": 500, "acc": 0.736, "auroc": 0.8451, "yes": 0.3, "gold_yes": 0.5}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 500

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · English baseline · aegis2 · yes

Solar Decide Flash, same named hosted route; English baseline; slice aegis2. {}

Source observations: {"n": 500, "acc": 0.736, "auroc": 0.8451, "yes": 0.3, "gold_yes": 0.5}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 500

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · English baseline · aegis2_response · acc

Solar Decide Flash, same named hosted route; English baseline; slice aegis2_response. {}

Source observations: {"n": 500, "acc": 0.72, "auroc": 0.8647, "yes": 0.28, "gold_yes": 0.5}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 500

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · English baseline · aegis2_response · auroc

Solar Decide Flash, same named hosted route; English baseline; slice aegis2_response. {}

Source observations: {"n": 500, "acc": 0.72, "auroc": 0.8647, "yes": 0.28, "gold_yes": 0.5}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 500

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · English baseline · aegis2_response · yes

Solar Decide Flash, same named hosted route; English baseline; slice aegis2_response. {}

Source observations: {"n": 500, "acc": 0.72, "auroc": 0.8647, "yes": 0.28, "gold_yes": 0.5}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 500

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · English baseline · atbench500 · acc

Solar Decide Flash, same named hosted route; English baseline; slice atbench500. {}

Source observations: {"n": 500, "acc": 0.5, "auroc": 0.6266, "yes": 0.864, "gold_yes": 0.5}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 500

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · English baseline · atbench500 · auroc

Solar Decide Flash, same named hosted route; English baseline; slice atbench500. {}

Source observations: {"n": 500, "acc": 0.5, "auroc": 0.6266, "yes": 0.864, "gold_yes": 0.5}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 500

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · English baseline · atbench500 · yes

Solar Decide Flash, same named hosted route; English baseline; slice atbench500. {}

Source observations: {"n": 500, "acc": 0.5, "auroc": 0.6266, "yes": 0.864, "gold_yes": 0.5}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 500

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · English baseline · jailbreak_classification · acc

Solar Decide Flash, same named hosted route; English baseline; slice jailbreak_classification. {}

Source observations: {"n": 400, "acc": 0.97, "auroc": 0.9883, "yes": 0.3525, "gold_yes": 0.3525}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 400

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · English baseline · jailbreak_classification · auroc

Solar Decide Flash, same named hosted route; English baseline; slice jailbreak_classification. {}

Source observations: {"n": 400, "acc": 0.97, "auroc": 0.9883, "yes": 0.3525, "gold_yes": 0.3525}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 400

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · English baseline · jailbreak_classification · yes

Solar Decide Flash, same named hosted route; English baseline; slice jailbreak_classification. {}

Source observations: {"n": 400, "acc": 0.97, "auroc": 0.9883, "yes": 0.3525, "gold_yes": 0.3525}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 400

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · English baseline · prompt_injections · acc

Solar Decide Flash, same named hosted route; English baseline; slice prompt_injections. {}

Source observations: {"n": 116, "acc": 0.7931, "auroc": 0.9051, "yes": 0.3966, "gold_yes": 0.5172}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 116

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · English baseline · prompt_injections · auroc

Solar Decide Flash, same named hosted route; English baseline; slice prompt_injections. {}

Source observations: {"n": 116, "acc": 0.7931, "auroc": 0.9051, "yes": 0.3966, "gold_yes": 0.5172}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 116

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · English baseline · prompt_injections · yes

Solar Decide Flash, same named hosted route; English baseline; slice prompt_injections. {}

Source observations: {"n": 116, "acc": 0.7931, "auroc": 0.9051, "yes": 0.3966, "gold_yes": 0.5172}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 116

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Chinese · overall · acc

Solar Decide Flash, same named hosted route; Chinese; slice overall. {"answered": 1979, "cases": 1979, "complete": true}

Source observations: {"n": 1979, "acc": 0.7032, "en_acc": 0.744, "delta": -0.0407, "ci": [-0.0615, -0.0194], "significant": true, "auroc": 0.7929, "en_auroc": 0.8474}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 1979

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Chinese · overall · auroc

Solar Decide Flash, same named hosted route; Chinese; slice overall. {"answered": 1979, "cases": 1979, "complete": true}

Source observations: {"n": 1979, "acc": 0.7032, "en_acc": 0.744, "delta": -0.0407, "ci": [-0.0615, -0.0194], "significant": true, "auroc": 0.7929, "en_auroc": 0.8474}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 1979

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Chinese · overall · en_acc

Solar Decide Flash, same named hosted route; Chinese; slice overall. {"answered": 1979, "cases": 1979, "complete": true}

Source observations: {"n": 1979, "acc": 0.7032, "en_acc": 0.744, "delta": -0.0407, "ci": [-0.0615, -0.0194], "significant": true, "auroc": 0.7929, "en_auroc": 0.8474}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 1979

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Chinese · overall · en_auroc

Solar Decide Flash, same named hosted route; Chinese; slice overall. {"answered": 1979, "cases": 1979, "complete": true}

Source observations: {"n": 1979, "acc": 0.7032, "en_acc": 0.744, "delta": -0.0407, "ci": [-0.0615, -0.0194], "significant": true, "auroc": 0.7929, "en_auroc": 0.8474}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 1979

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Chinese · overall · delta

Solar Decide Flash, same named hosted route; Chinese; slice overall. {"answered": 1979, "cases": 1979, "complete": true}

Source observations: {"n": 1979, "acc": 0.7032, "en_acc": 0.744, "delta": -0.0407, "ci": [-0.0615, -0.0194], "significant": true, "auroc": 0.7929, "en_auroc": 0.8474}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 1979

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Chinese · aegis2 · acc

Solar Decide Flash, same named hosted route; Chinese; slice aegis2. {"answered": 1979, "cases": 1979, "complete": true}

Source observations: {"n": 499, "acc": 0.6192, "en_acc": 0.7355, "auroc": 0.6606, "en_auroc": 0.846, "yes": 0.3447, "en_yes": 0.3006, "gold_yes": 0.501}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 499

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Chinese · aegis2 · auroc

Solar Decide Flash, same named hosted route; Chinese; slice aegis2. {"answered": 1979, "cases": 1979, "complete": true}

Source observations: {"n": 499, "acc": 0.6192, "en_acc": 0.7355, "auroc": 0.6606, "en_auroc": 0.846, "yes": 0.3447, "en_yes": 0.3006, "gold_yes": 0.501}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 499

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Chinese · aegis2 · en_acc

Solar Decide Flash, same named hosted route; Chinese; slice aegis2. {"answered": 1979, "cases": 1979, "complete": true}

Source observations: {"n": 499, "acc": 0.6192, "en_acc": 0.7355, "auroc": 0.6606, "en_auroc": 0.846, "yes": 0.3447, "en_yes": 0.3006, "gold_yes": 0.501}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 499

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Chinese · aegis2 · en_auroc

Solar Decide Flash, same named hosted route; Chinese; slice aegis2. {"answered": 1979, "cases": 1979, "complete": true}

Source observations: {"n": 499, "acc": 0.6192, "en_acc": 0.7355, "auroc": 0.6606, "en_auroc": 0.846, "yes": 0.3447, "en_yes": 0.3006, "gold_yes": 0.501}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 499

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Chinese · aegis2 · yes

Solar Decide Flash, same named hosted route; Chinese; slice aegis2. {"answered": 1979, "cases": 1979, "complete": true}

Source observations: {"n": 499, "acc": 0.6192, "en_acc": 0.7355, "auroc": 0.6606, "en_auroc": 0.846, "yes": 0.3447, "en_yes": 0.3006, "gold_yes": 0.501}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 499

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Chinese · aegis2 · en_yes

Solar Decide Flash, same named hosted route; Chinese; slice aegis2. {"answered": 1979, "cases": 1979, "complete": true}

Source observations: {"n": 499, "acc": 0.6192, "en_acc": 0.7355, "auroc": 0.6606, "en_auroc": 0.846, "yes": 0.3447, "en_yes": 0.3006, "gold_yes": 0.501}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 499

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Chinese · aegis2_response · acc

Solar Decide Flash, same named hosted route; Chinese; slice aegis2_response. {"answered": 1979, "cases": 1979, "complete": true}

Source observations: {"n": 500, "acc": 0.726, "en_acc": 0.72, "auroc": 0.8083, "en_auroc": 0.8647, "yes": 0.442, "en_yes": 0.28, "gold_yes": 0.5}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 500

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Chinese · aegis2_response · auroc

Solar Decide Flash, same named hosted route; Chinese; slice aegis2_response. {"answered": 1979, "cases": 1979, "complete": true}

Source observations: {"n": 500, "acc": 0.726, "en_acc": 0.72, "auroc": 0.8083, "en_auroc": 0.8647, "yes": 0.442, "en_yes": 0.28, "gold_yes": 0.5}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 500

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Chinese · aegis2_response · en_acc

Solar Decide Flash, same named hosted route; Chinese; slice aegis2_response. {"answered": 1979, "cases": 1979, "complete": true}

Source observations: {"n": 500, "acc": 0.726, "en_acc": 0.72, "auroc": 0.8083, "en_auroc": 0.8647, "yes": 0.442, "en_yes": 0.28, "gold_yes": 0.5}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 500

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Chinese · aegis2_response · en_auroc

Solar Decide Flash, same named hosted route; Chinese; slice aegis2_response. {"answered": 1979, "cases": 1979, "complete": true}

Source observations: {"n": 500, "acc": 0.726, "en_acc": 0.72, "auroc": 0.8083, "en_auroc": 0.8647, "yes": 0.442, "en_yes": 0.28, "gold_yes": 0.5}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 500

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Chinese · aegis2_response · yes

Solar Decide Flash, same named hosted route; Chinese; slice aegis2_response. {"answered": 1979, "cases": 1979, "complete": true}

Source observations: {"n": 500, "acc": 0.726, "en_acc": 0.72, "auroc": 0.8083, "en_auroc": 0.8647, "yes": 0.442, "en_yes": 0.28, "gold_yes": 0.5}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 500

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Chinese · aegis2_response · en_yes

Solar Decide Flash, same named hosted route; Chinese; slice aegis2_response. {"answered": 1979, "cases": 1979, "complete": true}

Source observations: {"n": 500, "acc": 0.726, "en_acc": 0.72, "auroc": 0.8083, "en_auroc": 0.8647, "yes": 0.442, "en_yes": 0.28, "gold_yes": 0.5}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 500

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Chinese · atbench500 · acc

Solar Decide Flash, same named hosted route; Chinese; slice atbench500. {"answered": 1979, "cases": 1979, "complete": true}

Source observations: {"n": 467, "acc": 0.4989, "en_acc": 0.4989, "auroc": 0.6284, "en_auroc": 0.6328, "yes": 0.985, "en_yes": 0.8608, "gold_yes": 0.4925}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 467

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Chinese · atbench500 · auroc

Solar Decide Flash, same named hosted route; Chinese; slice atbench500. {"answered": 1979, "cases": 1979, "complete": true}

Source observations: {"n": 467, "acc": 0.4989, "en_acc": 0.4989, "auroc": 0.6284, "en_auroc": 0.6328, "yes": 0.985, "en_yes": 0.8608, "gold_yes": 0.4925}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 467

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Chinese · atbench500 · en_acc

Solar Decide Flash, same named hosted route; Chinese; slice atbench500. {"answered": 1979, "cases": 1979, "complete": true}

Source observations: {"n": 467, "acc": 0.4989, "en_acc": 0.4989, "auroc": 0.6284, "en_auroc": 0.6328, "yes": 0.985, "en_yes": 0.8608, "gold_yes": 0.4925}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 467

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Chinese · atbench500 · en_auroc

Solar Decide Flash, same named hosted route; Chinese; slice atbench500. {"answered": 1979, "cases": 1979, "complete": true}

Source observations: {"n": 467, "acc": 0.4989, "en_acc": 0.4989, "auroc": 0.6284, "en_auroc": 0.6328, "yes": 0.985, "en_yes": 0.8608, "gold_yes": 0.4925}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 467

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Chinese · atbench500 · yes

Solar Decide Flash, same named hosted route; Chinese; slice atbench500. {"answered": 1979, "cases": 1979, "complete": true}

Source observations: {"n": 467, "acc": 0.4989, "en_acc": 0.4989, "auroc": 0.6284, "en_auroc": 0.6328, "yes": 0.985, "en_yes": 0.8608, "gold_yes": 0.4925}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 467

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Chinese · atbench500 · en_yes

Solar Decide Flash, same named hosted route; Chinese; slice atbench500. {"answered": 1979, "cases": 1979, "complete": true}

Source observations: {"n": 467, "acc": 0.4989, "en_acc": 0.4989, "auroc": 0.6284, "en_auroc": 0.6328, "yes": 0.985, "en_yes": 0.8608, "gold_yes": 0.4925}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 467

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Chinese · jailbreak_classification · acc

Solar Decide Flash, same named hosted route; Chinese; slice jailbreak_classification. {"answered": 1979, "cases": 1979, "complete": true}

Source observations: {"n": 397, "acc": 0.9824, "en_acc": 0.9723, "auroc": 0.9953, "en_auroc": 0.9884, "yes": 0.3451, "en_yes": 0.3501, "gold_yes": 0.3476}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 397

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Chinese · jailbreak_classification · auroc

Solar Decide Flash, same named hosted route; Chinese; slice jailbreak_classification. {"answered": 1979, "cases": 1979, "complete": true}

Source observations: {"n": 397, "acc": 0.9824, "en_acc": 0.9723, "auroc": 0.9953, "en_auroc": 0.9884, "yes": 0.3451, "en_yes": 0.3501, "gold_yes": 0.3476}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 397

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Chinese · jailbreak_classification · en_acc

Solar Decide Flash, same named hosted route; Chinese; slice jailbreak_classification. {"answered": 1979, "cases": 1979, "complete": true}

Source observations: {"n": 397, "acc": 0.9824, "en_acc": 0.9723, "auroc": 0.9953, "en_auroc": 0.9884, "yes": 0.3451, "en_yes": 0.3501, "gold_yes": 0.3476}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 397

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Chinese · jailbreak_classification · en_auroc

Solar Decide Flash, same named hosted route; Chinese; slice jailbreak_classification. {"answered": 1979, "cases": 1979, "complete": true}

Source observations: {"n": 397, "acc": 0.9824, "en_acc": 0.9723, "auroc": 0.9953, "en_auroc": 0.9884, "yes": 0.3451, "en_yes": 0.3501, "gold_yes": 0.3476}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 397

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Chinese · jailbreak_classification · yes

Solar Decide Flash, same named hosted route; Chinese; slice jailbreak_classification. {"answered": 1979, "cases": 1979, "complete": true}

Source observations: {"n": 397, "acc": 0.9824, "en_acc": 0.9723, "auroc": 0.9953, "en_auroc": 0.9884, "yes": 0.3451, "en_yes": 0.3501, "gold_yes": 0.3476}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 397

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Chinese · jailbreak_classification · en_yes

Solar Decide Flash, same named hosted route; Chinese; slice jailbreak_classification. {"answered": 1979, "cases": 1979, "complete": true}

Source observations: {"n": 397, "acc": 0.9824, "en_acc": 0.9723, "auroc": 0.9953, "en_auroc": 0.9884, "yes": 0.3451, "en_yes": 0.3501, "gold_yes": 0.3476}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 397

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Chinese · prompt_injections · acc

Solar Decide Flash, same named hosted route; Chinese; slice prompt_injections. {"answered": 1979, "cases": 1979, "complete": true}

Source observations: {"n": 116, "acc": 0.6897, "en_acc": 0.7931, "auroc": 0.872, "en_auroc": 0.9051, "yes": 0.2241, "en_yes": 0.3966, "gold_yes": 0.5172}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 116

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Chinese · prompt_injections · auroc

Solar Decide Flash, same named hosted route; Chinese; slice prompt_injections. {"answered": 1979, "cases": 1979, "complete": true}

Source observations: {"n": 116, "acc": 0.6897, "en_acc": 0.7931, "auroc": 0.872, "en_auroc": 0.9051, "yes": 0.2241, "en_yes": 0.3966, "gold_yes": 0.5172}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 116

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Chinese · prompt_injections · en_acc

Solar Decide Flash, same named hosted route; Chinese; slice prompt_injections. {"answered": 1979, "cases": 1979, "complete": true}

Source observations: {"n": 116, "acc": 0.6897, "en_acc": 0.7931, "auroc": 0.872, "en_auroc": 0.9051, "yes": 0.2241, "en_yes": 0.3966, "gold_yes": 0.5172}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 116

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Chinese · prompt_injections · en_auroc

Solar Decide Flash, same named hosted route; Chinese; slice prompt_injections. {"answered": 1979, "cases": 1979, "complete": true}

Source observations: {"n": 116, "acc": 0.6897, "en_acc": 0.7931, "auroc": 0.872, "en_auroc": 0.9051, "yes": 0.2241, "en_yes": 0.3966, "gold_yes": 0.5172}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 116

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Chinese · prompt_injections · yes

Solar Decide Flash, same named hosted route; Chinese; slice prompt_injections. {"answered": 1979, "cases": 1979, "complete": true}

Source observations: {"n": 116, "acc": 0.6897, "en_acc": 0.7931, "auroc": 0.872, "en_auroc": 0.9051, "yes": 0.2241, "en_yes": 0.3966, "gold_yes": 0.5172}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 116

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Chinese · prompt_injections · en_yes

Solar Decide Flash, same named hosted route; Chinese; slice prompt_injections. {"answered": 1979, "cases": 1979, "complete": true}

Source observations: {"n": 116, "acc": 0.6897, "en_acc": 0.7931, "auroc": 0.872, "en_auroc": 0.9051, "yes": 0.2241, "en_yes": 0.3966, "gold_yes": 0.5172}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 116

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Arabic · overall · acc

Solar Decide Flash, same named hosted route; Arabic; slice overall. {"answered": 2007, "cases": 2007, "complete": true}

Source observations: {"n": 2007, "acc": 0.6512, "en_acc": 0.7447, "delta": -0.0935, "ci": [-0.116, -0.0705], "significant": true, "auroc": 0.7938, "en_auroc": 0.8462}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 2007

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Arabic · overall · auroc

Solar Decide Flash, same named hosted route; Arabic; slice overall. {"answered": 2007, "cases": 2007, "complete": true}

Source observations: {"n": 2007, "acc": 0.6512, "en_acc": 0.7447, "delta": -0.0935, "ci": [-0.116, -0.0705], "significant": true, "auroc": 0.7938, "en_auroc": 0.8462}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 2007

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Arabic · overall · en_acc

Solar Decide Flash, same named hosted route; Arabic; slice overall. {"answered": 2007, "cases": 2007, "complete": true}

Source observations: {"n": 2007, "acc": 0.6512, "en_acc": 0.7447, "delta": -0.0935, "ci": [-0.116, -0.0705], "significant": true, "auroc": 0.7938, "en_auroc": 0.8462}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 2007

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Arabic · overall · en_auroc

Solar Decide Flash, same named hosted route; Arabic; slice overall. {"answered": 2007, "cases": 2007, "complete": true}

Source observations: {"n": 2007, "acc": 0.6512, "en_acc": 0.7447, "delta": -0.0935, "ci": [-0.116, -0.0705], "significant": true, "auroc": 0.7938, "en_auroc": 0.8462}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 2007

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Arabic · overall · delta

Solar Decide Flash, same named hosted route; Arabic; slice overall. {"answered": 2007, "cases": 2007, "complete": true}

Source observations: {"n": 2007, "acc": 0.6512, "en_acc": 0.7447, "delta": -0.0935, "ci": [-0.116, -0.0705], "significant": true, "auroc": 0.7938, "en_auroc": 0.8462}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 2007

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Arabic · aegis2 · acc

Solar Decide Flash, same named hosted route; Arabic; slice aegis2. {"answered": 2007, "cases": 2007, "complete": true}

Source observations: {"n": 500, "acc": 0.69, "en_acc": 0.736, "auroc": 0.8135, "en_auroc": 0.8451, "yes": 0.226, "en_yes": 0.3, "gold_yes": 0.5}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 500

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Arabic · aegis2 · auroc

Solar Decide Flash, same named hosted route; Arabic; slice aegis2. {"answered": 2007, "cases": 2007, "complete": true}

Source observations: {"n": 500, "acc": 0.69, "en_acc": 0.736, "auroc": 0.8135, "en_auroc": 0.8451, "yes": 0.226, "en_yes": 0.3, "gold_yes": 0.5}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 500

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Arabic · aegis2 · en_acc

Solar Decide Flash, same named hosted route; Arabic; slice aegis2. {"answered": 2007, "cases": 2007, "complete": true}

Source observations: {"n": 500, "acc": 0.69, "en_acc": 0.736, "auroc": 0.8135, "en_auroc": 0.8451, "yes": 0.226, "en_yes": 0.3, "gold_yes": 0.5}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 500

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Arabic · aegis2 · en_auroc

Solar Decide Flash, same named hosted route; Arabic; slice aegis2. {"answered": 2007, "cases": 2007, "complete": true}

Source observations: {"n": 500, "acc": 0.69, "en_acc": 0.736, "auroc": 0.8135, "en_auroc": 0.8451, "yes": 0.226, "en_yes": 0.3, "gold_yes": 0.5}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 500

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Arabic · aegis2 · yes

Solar Decide Flash, same named hosted route; Arabic; slice aegis2. {"answered": 2007, "cases": 2007, "complete": true}

Source observations: {"n": 500, "acc": 0.69, "en_acc": 0.736, "auroc": 0.8135, "en_auroc": 0.8451, "yes": 0.226, "en_yes": 0.3, "gold_yes": 0.5}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 500

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Arabic · aegis2 · en_yes

Solar Decide Flash, same named hosted route; Arabic; slice aegis2. {"answered": 2007, "cases": 2007, "complete": true}

Source observations: {"n": 500, "acc": 0.69, "en_acc": 0.736, "auroc": 0.8135, "en_auroc": 0.8451, "yes": 0.226, "en_yes": 0.3, "gold_yes": 0.5}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 500

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Arabic · aegis2_response · acc

Solar Decide Flash, same named hosted route; Arabic; slice aegis2_response. {"answered": 2007, "cases": 2007, "complete": true}

Source observations: {"n": 499, "acc": 0.7555, "en_acc": 0.7194, "auroc": 0.8242, "en_auroc": 0.8642, "yes": 0.3747, "en_yes": 0.2786, "gold_yes": 0.499}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 499

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Arabic · aegis2_response · auroc

Solar Decide Flash, same named hosted route; Arabic; slice aegis2_response. {"answered": 2007, "cases": 2007, "complete": true}

Source observations: {"n": 499, "acc": 0.7555, "en_acc": 0.7194, "auroc": 0.8242, "en_auroc": 0.8642, "yes": 0.3747, "en_yes": 0.2786, "gold_yes": 0.499}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 499

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Arabic · aegis2_response · en_acc

Solar Decide Flash, same named hosted route; Arabic; slice aegis2_response. {"answered": 2007, "cases": 2007, "complete": true}

Source observations: {"n": 499, "acc": 0.7555, "en_acc": 0.7194, "auroc": 0.8242, "en_auroc": 0.8642, "yes": 0.3747, "en_yes": 0.2786, "gold_yes": 0.499}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 499

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Arabic · aegis2_response · en_auroc

Solar Decide Flash, same named hosted route; Arabic; slice aegis2_response. {"answered": 2007, "cases": 2007, "complete": true}

Source observations: {"n": 499, "acc": 0.7555, "en_acc": 0.7194, "auroc": 0.8242, "en_auroc": 0.8642, "yes": 0.3747, "en_yes": 0.2786, "gold_yes": 0.499}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 499

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Arabic · aegis2_response · yes

Solar Decide Flash, same named hosted route; Arabic; slice aegis2_response. {"answered": 2007, "cases": 2007, "complete": true}

Source observations: {"n": 499, "acc": 0.7555, "en_acc": 0.7194, "auroc": 0.8242, "en_auroc": 0.8642, "yes": 0.3747, "en_yes": 0.2786, "gold_yes": 0.499}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 499

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Arabic · aegis2_response · en_yes

Solar Decide Flash, same named hosted route; Arabic; slice aegis2_response. {"answered": 2007, "cases": 2007, "complete": true}

Source observations: {"n": 499, "acc": 0.7555, "en_acc": 0.7194, "auroc": 0.8242, "en_auroc": 0.8642, "yes": 0.3747, "en_yes": 0.2786, "gold_yes": 0.499}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 499

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Arabic · atbench500 · acc

Solar Decide Flash, same named hosted route; Arabic; slice atbench500. {"answered": 2007, "cases": 2007, "complete": true}

Source observations: {"n": 493, "acc": 0.499, "en_acc": 0.5051, "auroc": 0.5409, "en_auroc": 0.6283, "yes": 0.9736, "en_yes": 0.8621, "gold_yes": 0.5051}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 493

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Arabic · atbench500 · auroc

Solar Decide Flash, same named hosted route; Arabic; slice atbench500. {"answered": 2007, "cases": 2007, "complete": true}

Source observations: {"n": 493, "acc": 0.499, "en_acc": 0.5051, "auroc": 0.5409, "en_auroc": 0.6283, "yes": 0.9736, "en_yes": 0.8621, "gold_yes": 0.5051}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 493

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Arabic · atbench500 · en_acc

Solar Decide Flash, same named hosted route; Arabic; slice atbench500. {"answered": 2007, "cases": 2007, "complete": true}

Source observations: {"n": 493, "acc": 0.499, "en_acc": 0.5051, "auroc": 0.5409, "en_auroc": 0.6283, "yes": 0.9736, "en_yes": 0.8621, "gold_yes": 0.5051}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 493

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Arabic · atbench500 · en_auroc

Solar Decide Flash, same named hosted route; Arabic; slice atbench500. {"answered": 2007, "cases": 2007, "complete": true}

Source observations: {"n": 493, "acc": 0.499, "en_acc": 0.5051, "auroc": 0.5409, "en_auroc": 0.6283, "yes": 0.9736, "en_yes": 0.8621, "gold_yes": 0.5051}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 493

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Arabic · atbench500 · yes

Solar Decide Flash, same named hosted route; Arabic; slice atbench500. {"answered": 2007, "cases": 2007, "complete": true}

Source observations: {"n": 493, "acc": 0.499, "en_acc": 0.5051, "auroc": 0.5409, "en_auroc": 0.6283, "yes": 0.9736, "en_yes": 0.8621, "gold_yes": 0.5051}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 493

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Arabic · atbench500 · en_yes

Solar Decide Flash, same named hosted route; Arabic; slice atbench500. {"answered": 2007, "cases": 2007, "complete": true}

Source observations: {"n": 493, "acc": 0.499, "en_acc": 0.5051, "auroc": 0.5409, "en_auroc": 0.6283, "yes": 0.9736, "en_yes": 0.8621, "gold_yes": 0.5051}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 493

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Arabic · jailbreak_classification · acc

Solar Decide Flash, same named hosted route; Arabic; slice jailbreak_classification. {"answered": 2007, "cases": 2007, "complete": true}

Source observations: {"n": 399, "acc": 0.7168, "en_acc": 0.9699, "auroc": 0.988, "en_auroc": 0.9883, "yes": 0.6291, "en_yes": 0.3509, "gold_yes": 0.3509}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 399

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Arabic · jailbreak_classification · auroc

Solar Decide Flash, same named hosted route; Arabic; slice jailbreak_classification. {"answered": 2007, "cases": 2007, "complete": true}

Source observations: {"n": 399, "acc": 0.7168, "en_acc": 0.9699, "auroc": 0.988, "en_auroc": 0.9883, "yes": 0.6291, "en_yes": 0.3509, "gold_yes": 0.3509}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 399

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Arabic · jailbreak_classification · en_acc

Solar Decide Flash, same named hosted route; Arabic; slice jailbreak_classification. {"answered": 2007, "cases": 2007, "complete": true}

Source observations: {"n": 399, "acc": 0.7168, "en_acc": 0.9699, "auroc": 0.988, "en_auroc": 0.9883, "yes": 0.6291, "en_yes": 0.3509, "gold_yes": 0.3509}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 399

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Arabic · jailbreak_classification · en_auroc

Solar Decide Flash, same named hosted route; Arabic; slice jailbreak_classification. {"answered": 2007, "cases": 2007, "complete": true}

Source observations: {"n": 399, "acc": 0.7168, "en_acc": 0.9699, "auroc": 0.988, "en_auroc": 0.9883, "yes": 0.6291, "en_yes": 0.3509, "gold_yes": 0.3509}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 399

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Arabic · jailbreak_classification · yes

Solar Decide Flash, same named hosted route; Arabic; slice jailbreak_classification. {"answered": 2007, "cases": 2007, "complete": true}

Source observations: {"n": 399, "acc": 0.7168, "en_acc": 0.9699, "auroc": 0.988, "en_auroc": 0.9883, "yes": 0.6291, "en_yes": 0.3509, "gold_yes": 0.3509}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 399

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Arabic · jailbreak_classification · en_yes

Solar Decide Flash, same named hosted route; Arabic; slice jailbreak_classification. {"answered": 2007, "cases": 2007, "complete": true}

Source observations: {"n": 399, "acc": 0.7168, "en_acc": 0.9699, "auroc": 0.988, "en_auroc": 0.9883, "yes": 0.6291, "en_yes": 0.3509, "gold_yes": 0.3509}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 399

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Arabic · prompt_injections · acc

Solar Decide Flash, same named hosted route; Arabic; slice prompt_injections. {"answered": 2007, "cases": 2007, "complete": true}

Source observations: {"n": 116, "acc": 0.5948, "en_acc": 0.7931, "auroc": 0.8024, "en_auroc": 0.9051, "yes": 0.1121, "en_yes": 0.3966, "gold_yes": 0.5172}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 116

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Arabic · prompt_injections · auroc

Solar Decide Flash, same named hosted route; Arabic; slice prompt_injections. {"answered": 2007, "cases": 2007, "complete": true}

Source observations: {"n": 116, "acc": 0.5948, "en_acc": 0.7931, "auroc": 0.8024, "en_auroc": 0.9051, "yes": 0.1121, "en_yes": 0.3966, "gold_yes": 0.5172}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 116

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Arabic · prompt_injections · en_acc

Solar Decide Flash, same named hosted route; Arabic; slice prompt_injections. {"answered": 2007, "cases": 2007, "complete": true}

Source observations: {"n": 116, "acc": 0.5948, "en_acc": 0.7931, "auroc": 0.8024, "en_auroc": 0.9051, "yes": 0.1121, "en_yes": 0.3966, "gold_yes": 0.5172}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 116

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Arabic · prompt_injections · en_auroc

Solar Decide Flash, same named hosted route; Arabic; slice prompt_injections. {"answered": 2007, "cases": 2007, "complete": true}

Source observations: {"n": 116, "acc": 0.5948, "en_acc": 0.7931, "auroc": 0.8024, "en_auroc": 0.9051, "yes": 0.1121, "en_yes": 0.3966, "gold_yes": 0.5172}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 116

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Arabic · prompt_injections · yes

Solar Decide Flash, same named hosted route; Arabic; slice prompt_injections. {"answered": 2007, "cases": 2007, "complete": true}

Source observations: {"n": 116, "acc": 0.5948, "en_acc": 0.7931, "auroc": 0.8024, "en_auroc": 0.9051, "yes": 0.1121, "en_yes": 0.3966, "gold_yes": 0.5172}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 116

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Arabic · prompt_injections · en_yes

Solar Decide Flash, same named hosted route; Arabic; slice prompt_injections. {"answered": 2007, "cases": 2007, "complete": true}

Source observations: {"n": 116, "acc": 0.5948, "en_acc": 0.7931, "auroc": 0.8024, "en_auroc": 0.9051, "yes": 0.1121, "en_yes": 0.3966, "gold_yes": 0.5172}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 116

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Hindi · overall · acc

Solar Decide Flash, same named hosted route; Hindi; slice overall. {"answered": 1999, "cases": 1999, "complete": true}

Source observations: {"n": 1999, "acc": 0.6878, "en_acc": 0.7452, "delta": -0.0574, "ci": [-0.0802, -0.0347], "significant": true, "auroc": 0.7832, "en_auroc": 0.8463}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 1999

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Hindi · overall · auroc

Solar Decide Flash, same named hosted route; Hindi; slice overall. {"answered": 1999, "cases": 1999, "complete": true}

Source observations: {"n": 1999, "acc": 0.6878, "en_acc": 0.7452, "delta": -0.0574, "ci": [-0.0802, -0.0347], "significant": true, "auroc": 0.7832, "en_auroc": 0.8463}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 1999

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Hindi · overall · en_acc

Solar Decide Flash, same named hosted route; Hindi; slice overall. {"answered": 1999, "cases": 1999, "complete": true}

Source observations: {"n": 1999, "acc": 0.6878, "en_acc": 0.7452, "delta": -0.0574, "ci": [-0.0802, -0.0347], "significant": true, "auroc": 0.7832, "en_auroc": 0.8463}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 1999

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Hindi · overall · en_auroc

Solar Decide Flash, same named hosted route; Hindi; slice overall. {"answered": 1999, "cases": 1999, "complete": true}

Source observations: {"n": 1999, "acc": 0.6878, "en_acc": 0.7452, "delta": -0.0574, "ci": [-0.0802, -0.0347], "significant": true, "auroc": 0.7832, "en_auroc": 0.8463}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 1999

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Hindi · overall · delta

Solar Decide Flash, same named hosted route; Hindi; slice overall. {"answered": 1999, "cases": 1999, "complete": true}

Source observations: {"n": 1999, "acc": 0.6878, "en_acc": 0.7452, "delta": -0.0574, "ci": [-0.0802, -0.0347], "significant": true, "auroc": 0.7832, "en_auroc": 0.8463}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 1999

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Hindi · aegis2 · acc

Solar Decide Flash, same named hosted route; Hindi; slice aegis2. {"answered": 1999, "cases": 1999, "complete": true}

Source observations: {"n": 500, "acc": 0.598, "en_acc": 0.736, "auroc": 0.7348, "en_auroc": 0.8451, "yes": 0.126, "en_yes": 0.3, "gold_yes": 0.5}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 500

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Hindi · aegis2 · auroc

Solar Decide Flash, same named hosted route; Hindi; slice aegis2. {"answered": 1999, "cases": 1999, "complete": true}

Source observations: {"n": 500, "acc": 0.598, "en_acc": 0.736, "auroc": 0.7348, "en_auroc": 0.8451, "yes": 0.126, "en_yes": 0.3, "gold_yes": 0.5}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 500

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Hindi · aegis2 · en_acc

Solar Decide Flash, same named hosted route; Hindi; slice aegis2. {"answered": 1999, "cases": 1999, "complete": true}

Source observations: {"n": 500, "acc": 0.598, "en_acc": 0.736, "auroc": 0.7348, "en_auroc": 0.8451, "yes": 0.126, "en_yes": 0.3, "gold_yes": 0.5}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 500

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Hindi · aegis2 · en_auroc

Solar Decide Flash, same named hosted route; Hindi; slice aegis2. {"answered": 1999, "cases": 1999, "complete": true}

Source observations: {"n": 500, "acc": 0.598, "en_acc": 0.736, "auroc": 0.7348, "en_auroc": 0.8451, "yes": 0.126, "en_yes": 0.3, "gold_yes": 0.5}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 500

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Hindi · aegis2 · yes

Solar Decide Flash, same named hosted route; Hindi; slice aegis2. {"answered": 1999, "cases": 1999, "complete": true}

Source observations: {"n": 500, "acc": 0.598, "en_acc": 0.736, "auroc": 0.7348, "en_auroc": 0.8451, "yes": 0.126, "en_yes": 0.3, "gold_yes": 0.5}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 500

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Hindi · aegis2 · en_yes

Solar Decide Flash, same named hosted route; Hindi; slice aegis2. {"answered": 1999, "cases": 1999, "complete": true}

Source observations: {"n": 500, "acc": 0.598, "en_acc": 0.736, "auroc": 0.7348, "en_auroc": 0.8451, "yes": 0.126, "en_yes": 0.3, "gold_yes": 0.5}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 500

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Hindi · aegis2_response · acc

Solar Decide Flash, same named hosted route; Hindi; slice aegis2_response. {"answered": 1999, "cases": 1999, "complete": true}

Source observations: {"n": 499, "acc": 0.6453, "en_acc": 0.7194, "auroc": 0.7121, "en_auroc": 0.8642, "yes": 0.3327, "en_yes": 0.2786, "gold_yes": 0.499}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 499

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Hindi · aegis2_response · auroc

Solar Decide Flash, same named hosted route; Hindi; slice aegis2_response. {"answered": 1999, "cases": 1999, "complete": true}

Source observations: {"n": 499, "acc": 0.6453, "en_acc": 0.7194, "auroc": 0.7121, "en_auroc": 0.8642, "yes": 0.3327, "en_yes": 0.2786, "gold_yes": 0.499}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 499

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Hindi · aegis2_response · en_acc

Solar Decide Flash, same named hosted route; Hindi; slice aegis2_response. {"answered": 1999, "cases": 1999, "complete": true}

Source observations: {"n": 499, "acc": 0.6453, "en_acc": 0.7194, "auroc": 0.7121, "en_auroc": 0.8642, "yes": 0.3327, "en_yes": 0.2786, "gold_yes": 0.499}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 499

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Hindi · aegis2_response · en_auroc

Solar Decide Flash, same named hosted route; Hindi; slice aegis2_response. {"answered": 1999, "cases": 1999, "complete": true}

Source observations: {"n": 499, "acc": 0.6453, "en_acc": 0.7194, "auroc": 0.7121, "en_auroc": 0.8642, "yes": 0.3327, "en_yes": 0.2786, "gold_yes": 0.499}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 499

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Hindi · aegis2_response · yes

Solar Decide Flash, same named hosted route; Hindi; slice aegis2_response. {"answered": 1999, "cases": 1999, "complete": true}

Source observations: {"n": 499, "acc": 0.6453, "en_acc": 0.7194, "auroc": 0.7121, "en_auroc": 0.8642, "yes": 0.3327, "en_yes": 0.2786, "gold_yes": 0.499}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 499

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Hindi · aegis2_response · en_yes

Solar Decide Flash, same named hosted route; Hindi; slice aegis2_response. {"answered": 1999, "cases": 1999, "complete": true}

Source observations: {"n": 499, "acc": 0.6453, "en_acc": 0.7194, "auroc": 0.7121, "en_auroc": 0.8642, "yes": 0.3327, "en_yes": 0.2786, "gold_yes": 0.499}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 499

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Hindi · atbench500 · acc

Solar Decide Flash, same named hosted route; Hindi; slice atbench500. {"answered": 1999, "cases": 1999, "complete": true}

Source observations: {"n": 487, "acc": 0.4743, "en_acc": 0.5051, "auroc": 0.5684, "en_auroc": 0.6289, "yes": 0.9487, "en_yes": 0.8604, "gold_yes": 0.5051}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 487

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Hindi · atbench500 · auroc

Solar Decide Flash, same named hosted route; Hindi; slice atbench500. {"answered": 1999, "cases": 1999, "complete": true}

Source observations: {"n": 487, "acc": 0.4743, "en_acc": 0.5051, "auroc": 0.5684, "en_auroc": 0.6289, "yes": 0.9487, "en_yes": 0.8604, "gold_yes": 0.5051}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 487

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Hindi · atbench500 · en_acc

Solar Decide Flash, same named hosted route; Hindi; slice atbench500. {"answered": 1999, "cases": 1999, "complete": true}

Source observations: {"n": 487, "acc": 0.4743, "en_acc": 0.5051, "auroc": 0.5684, "en_auroc": 0.6289, "yes": 0.9487, "en_yes": 0.8604, "gold_yes": 0.5051}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 487

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Hindi · atbench500 · en_auroc

Solar Decide Flash, same named hosted route; Hindi; slice atbench500. {"answered": 1999, "cases": 1999, "complete": true}

Source observations: {"n": 487, "acc": 0.4743, "en_acc": 0.5051, "auroc": 0.5684, "en_auroc": 0.6289, "yes": 0.9487, "en_yes": 0.8604, "gold_yes": 0.5051}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 487

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Hindi · atbench500 · yes

Solar Decide Flash, same named hosted route; Hindi; slice atbench500. {"answered": 1999, "cases": 1999, "complete": true}

Source observations: {"n": 487, "acc": 0.4743, "en_acc": 0.5051, "auroc": 0.5684, "en_auroc": 0.6289, "yes": 0.9487, "en_yes": 0.8604, "gold_yes": 0.5051}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 487

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Hindi · atbench500 · en_yes

Solar Decide Flash, same named hosted route; Hindi; slice atbench500. {"answered": 1999, "cases": 1999, "complete": true}

Source observations: {"n": 487, "acc": 0.4743, "en_acc": 0.5051, "auroc": 0.5684, "en_auroc": 0.6289, "yes": 0.9487, "en_yes": 0.8604, "gold_yes": 0.5051}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 487

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Hindi · jailbreak_classification · acc

Solar Decide Flash, same named hosted route; Hindi; slice jailbreak_classification. {"answered": 1999, "cases": 1999, "complete": true}

Source observations: {"n": 397, "acc": 0.937, "en_acc": 0.9723, "auroc": 0.9853, "en_auroc": 0.9884, "yes": 0.3451, "en_yes": 0.3501, "gold_yes": 0.3476}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 397

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Hindi · jailbreak_classification · auroc

Solar Decide Flash, same named hosted route; Hindi; slice jailbreak_classification. {"answered": 1999, "cases": 1999, "complete": true}

Source observations: {"n": 397, "acc": 0.937, "en_acc": 0.9723, "auroc": 0.9853, "en_auroc": 0.9884, "yes": 0.3451, "en_yes": 0.3501, "gold_yes": 0.3476}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 397

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Hindi · jailbreak_classification · en_acc

Solar Decide Flash, same named hosted route; Hindi; slice jailbreak_classification. {"answered": 1999, "cases": 1999, "complete": true}

Source observations: {"n": 397, "acc": 0.937, "en_acc": 0.9723, "auroc": 0.9853, "en_auroc": 0.9884, "yes": 0.3451, "en_yes": 0.3501, "gold_yes": 0.3476}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 397

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Hindi · jailbreak_classification · en_auroc

Solar Decide Flash, same named hosted route; Hindi; slice jailbreak_classification. {"answered": 1999, "cases": 1999, "complete": true}

Source observations: {"n": 397, "acc": 0.937, "en_acc": 0.9723, "auroc": 0.9853, "en_auroc": 0.9884, "yes": 0.3451, "en_yes": 0.3501, "gold_yes": 0.3476}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 397

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Hindi · jailbreak_classification · yes

Solar Decide Flash, same named hosted route; Hindi; slice jailbreak_classification. {"answered": 1999, "cases": 1999, "complete": true}

Source observations: {"n": 397, "acc": 0.937, "en_acc": 0.9723, "auroc": 0.9853, "en_auroc": 0.9884, "yes": 0.3451, "en_yes": 0.3501, "gold_yes": 0.3476}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 397

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Hindi · jailbreak_classification · en_yes

Solar Decide Flash, same named hosted route; Hindi; slice jailbreak_classification. {"answered": 1999, "cases": 1999, "complete": true}

Source observations: {"n": 397, "acc": 0.937, "en_acc": 0.9723, "auroc": 0.9853, "en_auroc": 0.9884, "yes": 0.3451, "en_yes": 0.3501, "gold_yes": 0.3476}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 397

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Hindi · prompt_injections · acc

Solar Decide Flash, same named hosted route; Hindi; slice prompt_injections. {"answered": 1999, "cases": 1999, "complete": true}

Source observations: {"n": 116, "acc": 0.7845, "en_acc": 0.7931, "auroc": 0.9152, "en_auroc": 0.9051, "yes": 0.319, "en_yes": 0.3966, "gold_yes": 0.5172}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 116

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Hindi · prompt_injections · auroc

Solar Decide Flash, same named hosted route; Hindi; slice prompt_injections. {"answered": 1999, "cases": 1999, "complete": true}

Source observations: {"n": 116, "acc": 0.7845, "en_acc": 0.7931, "auroc": 0.9152, "en_auroc": 0.9051, "yes": 0.319, "en_yes": 0.3966, "gold_yes": 0.5172}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 116

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Hindi · prompt_injections · en_acc

Solar Decide Flash, same named hosted route; Hindi; slice prompt_injections. {"answered": 1999, "cases": 1999, "complete": true}

Source observations: {"n": 116, "acc": 0.7845, "en_acc": 0.7931, "auroc": 0.9152, "en_auroc": 0.9051, "yes": 0.319, "en_yes": 0.3966, "gold_yes": 0.5172}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 116

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Hindi · prompt_injections · en_auroc

Solar Decide Flash, same named hosted route; Hindi; slice prompt_injections. {"answered": 1999, "cases": 1999, "complete": true}

Source observations: {"n": 116, "acc": 0.7845, "en_acc": 0.7931, "auroc": 0.9152, "en_auroc": 0.9051, "yes": 0.319, "en_yes": 0.3966, "gold_yes": 0.5172}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 116

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Hindi · prompt_injections · yes

Solar Decide Flash, same named hosted route; Hindi; slice prompt_injections. {"answered": 1999, "cases": 1999, "complete": true}

Source observations: {"n": 116, "acc": 0.7845, "en_acc": 0.7931, "auroc": 0.9152, "en_auroc": 0.9051, "yes": 0.319, "en_yes": 0.3966, "gold_yes": 0.5172}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 116

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Hindi · prompt_injections · en_yes

Solar Decide Flash, same named hosted route; Hindi; slice prompt_injections. {"answered": 1999, "cases": 1999, "complete": true}

Source observations: {"n": 116, "acc": 0.7845, "en_acc": 0.7931, "auroc": 0.9152, "en_auroc": 0.9051, "yes": 0.319, "en_yes": 0.3966, "gold_yes": 0.5172}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 116

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Vietnamese · overall · acc

Solar Decide Flash, same named hosted route; Vietnamese; slice overall. {"answered": 1968, "cases": 1968, "complete": true}

Source observations: {"n": 1968, "acc": 0.701, "en_acc": 0.7492, "delta": -0.0482, "ci": [-0.0716, -0.0257], "significant": true, "auroc": 0.784, "en_auroc": 0.8464}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 1968

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Vietnamese · overall · auroc

Solar Decide Flash, same named hosted route; Vietnamese; slice overall. {"answered": 1968, "cases": 1968, "complete": true}

Source observations: {"n": 1968, "acc": 0.701, "en_acc": 0.7492, "delta": -0.0482, "ci": [-0.0716, -0.0257], "significant": true, "auroc": 0.784, "en_auroc": 0.8464}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 1968

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Vietnamese · overall · en_acc

Solar Decide Flash, same named hosted route; Vietnamese; slice overall. {"answered": 1968, "cases": 1968, "complete": true}

Source observations: {"n": 1968, "acc": 0.701, "en_acc": 0.7492, "delta": -0.0482, "ci": [-0.0716, -0.0257], "significant": true, "auroc": 0.784, "en_auroc": 0.8464}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 1968

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Vietnamese · overall · en_auroc

Solar Decide Flash, same named hosted route; Vietnamese; slice overall. {"answered": 1968, "cases": 1968, "complete": true}

Source observations: {"n": 1968, "acc": 0.701, "en_acc": 0.7492, "delta": -0.0482, "ci": [-0.0716, -0.0257], "significant": true, "auroc": 0.784, "en_auroc": 0.8464}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 1968

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Vietnamese · overall · delta

Solar Decide Flash, same named hosted route; Vietnamese; slice overall. {"answered": 1968, "cases": 1968, "complete": true}

Source observations: {"n": 1968, "acc": 0.701, "en_acc": 0.7492, "delta": -0.0482, "ci": [-0.0716, -0.0257], "significant": true, "auroc": 0.784, "en_auroc": 0.8464}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 1968

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Vietnamese · aegis2 · acc

Solar Decide Flash, same named hosted route; Vietnamese; slice aegis2. {"answered": 1968, "cases": 1968, "complete": true}

Source observations: {"n": 497, "acc": 0.66, "en_acc": 0.7384, "auroc": 0.7908, "en_auroc": 0.8441, "yes": 0.1851, "en_yes": 0.2998, "gold_yes": 0.497}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 497

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Vietnamese · aegis2 · auroc

Solar Decide Flash, same named hosted route; Vietnamese; slice aegis2. {"answered": 1968, "cases": 1968, "complete": true}

Source observations: {"n": 497, "acc": 0.66, "en_acc": 0.7384, "auroc": 0.7908, "en_auroc": 0.8441, "yes": 0.1851, "en_yes": 0.2998, "gold_yes": 0.497}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 497

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Vietnamese · aegis2 · en_acc

Solar Decide Flash, same named hosted route; Vietnamese; slice aegis2. {"answered": 1968, "cases": 1968, "complete": true}

Source observations: {"n": 497, "acc": 0.66, "en_acc": 0.7384, "auroc": 0.7908, "en_auroc": 0.8441, "yes": 0.1851, "en_yes": 0.2998, "gold_yes": 0.497}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 497

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Vietnamese · aegis2 · en_auroc

Solar Decide Flash, same named hosted route; Vietnamese; slice aegis2. {"answered": 1968, "cases": 1968, "complete": true}

Source observations: {"n": 497, "acc": 0.66, "en_acc": 0.7384, "auroc": 0.7908, "en_auroc": 0.8441, "yes": 0.1851, "en_yes": 0.2998, "gold_yes": 0.497}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 497

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Vietnamese · aegis2 · yes

Solar Decide Flash, same named hosted route; Vietnamese; slice aegis2. {"answered": 1968, "cases": 1968, "complete": true}

Source observations: {"n": 497, "acc": 0.66, "en_acc": 0.7384, "auroc": 0.7908, "en_auroc": 0.8441, "yes": 0.1851, "en_yes": 0.2998, "gold_yes": 0.497}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 497

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Vietnamese · aegis2 · en_yes

Solar Decide Flash, same named hosted route; Vietnamese; slice aegis2. {"answered": 1968, "cases": 1968, "complete": true}

Source observations: {"n": 497, "acc": 0.66, "en_acc": 0.7384, "auroc": 0.7908, "en_auroc": 0.8441, "yes": 0.1851, "en_yes": 0.2998, "gold_yes": 0.497}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 497

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Vietnamese · aegis2_response · acc

Solar Decide Flash, same named hosted route; Vietnamese; slice aegis2_response. {"answered": 1968, "cases": 1968, "complete": true}

Source observations: {"n": 499, "acc": 0.6673, "en_acc": 0.7214, "auroc": 0.7997, "en_auroc": 0.8645, "yes": 0.2465, "en_yes": 0.2806, "gold_yes": 0.499}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 499

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Vietnamese · aegis2_response · auroc

Solar Decide Flash, same named hosted route; Vietnamese; slice aegis2_response. {"answered": 1968, "cases": 1968, "complete": true}

Source observations: {"n": 499, "acc": 0.6673, "en_acc": 0.7214, "auroc": 0.7997, "en_auroc": 0.8645, "yes": 0.2465, "en_yes": 0.2806, "gold_yes": 0.499}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 499

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Vietnamese · aegis2_response · en_acc

Solar Decide Flash, same named hosted route; Vietnamese; slice aegis2_response. {"answered": 1968, "cases": 1968, "complete": true}

Source observations: {"n": 499, "acc": 0.6673, "en_acc": 0.7214, "auroc": 0.7997, "en_auroc": 0.8645, "yes": 0.2465, "en_yes": 0.2806, "gold_yes": 0.499}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 499

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Vietnamese · aegis2_response · en_auroc

Solar Decide Flash, same named hosted route; Vietnamese; slice aegis2_response. {"answered": 1968, "cases": 1968, "complete": true}

Source observations: {"n": 499, "acc": 0.6673, "en_acc": 0.7214, "auroc": 0.7997, "en_auroc": 0.8645, "yes": 0.2465, "en_yes": 0.2806, "gold_yes": 0.499}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 499

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Vietnamese · aegis2_response · yes

Solar Decide Flash, same named hosted route; Vietnamese; slice aegis2_response. {"answered": 1968, "cases": 1968, "complete": true}

Source observations: {"n": 499, "acc": 0.6673, "en_acc": 0.7214, "auroc": 0.7997, "en_auroc": 0.8645, "yes": 0.2465, "en_yes": 0.2806, "gold_yes": 0.499}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 499

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Vietnamese · aegis2_response · en_yes

Solar Decide Flash, same named hosted route; Vietnamese; slice aegis2_response. {"answered": 1968, "cases": 1968, "complete": true}

Source observations: {"n": 499, "acc": 0.6673, "en_acc": 0.7214, "auroc": 0.7997, "en_auroc": 0.8645, "yes": 0.2465, "en_yes": 0.2806, "gold_yes": 0.499}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 499

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Vietnamese · atbench500 · acc

Solar Decide Flash, same named hosted route; Vietnamese; slice atbench500. {"answered": 1968, "cases": 1968, "complete": true}

Source observations: {"n": 463, "acc": 0.4946, "en_acc": 0.514, "auroc": 0.4993, "en_auroc": 0.6296, "yes": 0.9525, "en_yes": 0.8596, "gold_yes": 0.5119}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 463

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Vietnamese · atbench500 · auroc

Solar Decide Flash, same named hosted route; Vietnamese; slice atbench500. {"answered": 1968, "cases": 1968, "complete": true}

Source observations: {"n": 463, "acc": 0.4946, "en_acc": 0.514, "auroc": 0.4993, "en_auroc": 0.6296, "yes": 0.9525, "en_yes": 0.8596, "gold_yes": 0.5119}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 463

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Vietnamese · atbench500 · en_acc

Solar Decide Flash, same named hosted route; Vietnamese; slice atbench500. {"answered": 1968, "cases": 1968, "complete": true}

Source observations: {"n": 463, "acc": 0.4946, "en_acc": 0.514, "auroc": 0.4993, "en_auroc": 0.6296, "yes": 0.9525, "en_yes": 0.8596, "gold_yes": 0.5119}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 463

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Vietnamese · atbench500 · en_auroc

Solar Decide Flash, same named hosted route; Vietnamese; slice atbench500. {"answered": 1968, "cases": 1968, "complete": true}

Source observations: {"n": 463, "acc": 0.4946, "en_acc": 0.514, "auroc": 0.4993, "en_auroc": 0.6296, "yes": 0.9525, "en_yes": 0.8596, "gold_yes": 0.5119}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 463

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Vietnamese · atbench500 · yes

Solar Decide Flash, same named hosted route; Vietnamese; slice atbench500. {"answered": 1968, "cases": 1968, "complete": true}

Source observations: {"n": 463, "acc": 0.4946, "en_acc": 0.514, "auroc": 0.4993, "en_auroc": 0.6296, "yes": 0.9525, "en_yes": 0.8596, "gold_yes": 0.5119}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 463

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Vietnamese · atbench500 · en_yes

Solar Decide Flash, same named hosted route; Vietnamese; slice atbench500. {"answered": 1968, "cases": 1968, "complete": true}

Source observations: {"n": 463, "acc": 0.4946, "en_acc": 0.514, "auroc": 0.4993, "en_auroc": 0.6296, "yes": 0.9525, "en_yes": 0.8596, "gold_yes": 0.5119}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 463

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Vietnamese · jailbreak_classification · acc

Solar Decide Flash, same named hosted route; Vietnamese; slice jailbreak_classification. {"answered": 1968, "cases": 1968, "complete": true}

Source observations: {"n": 394, "acc": 0.9264, "en_acc": 0.9721, "auroc": 0.9788, "en_auroc": 0.9884, "yes": 0.4036, "en_yes": 0.3528, "gold_yes": 0.3503}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 394

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Vietnamese · jailbreak_classification · auroc

Solar Decide Flash, same named hosted route; Vietnamese; slice jailbreak_classification. {"answered": 1968, "cases": 1968, "complete": true}

Source observations: {"n": 394, "acc": 0.9264, "en_acc": 0.9721, "auroc": 0.9788, "en_auroc": 0.9884, "yes": 0.4036, "en_yes": 0.3528, "gold_yes": 0.3503}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 394

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Vietnamese · jailbreak_classification · en_acc

Solar Decide Flash, same named hosted route; Vietnamese; slice jailbreak_classification. {"answered": 1968, "cases": 1968, "complete": true}

Source observations: {"n": 394, "acc": 0.9264, "en_acc": 0.9721, "auroc": 0.9788, "en_auroc": 0.9884, "yes": 0.4036, "en_yes": 0.3528, "gold_yes": 0.3503}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 394

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Vietnamese · jailbreak_classification · en_auroc

Solar Decide Flash, same named hosted route; Vietnamese; slice jailbreak_classification. {"answered": 1968, "cases": 1968, "complete": true}

Source observations: {"n": 394, "acc": 0.9264, "en_acc": 0.9721, "auroc": 0.9788, "en_auroc": 0.9884, "yes": 0.4036, "en_yes": 0.3528, "gold_yes": 0.3503}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 394

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Vietnamese · jailbreak_classification · yes

Solar Decide Flash, same named hosted route; Vietnamese; slice jailbreak_classification. {"answered": 1968, "cases": 1968, "complete": true}

Source observations: {"n": 394, "acc": 0.9264, "en_acc": 0.9721, "auroc": 0.9788, "en_auroc": 0.9884, "yes": 0.4036, "en_yes": 0.3528, "gold_yes": 0.3503}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 394

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Vietnamese · jailbreak_classification · en_yes

Solar Decide Flash, same named hosted route; Vietnamese; slice jailbreak_classification. {"answered": 1968, "cases": 1968, "complete": true}

Source observations: {"n": 394, "acc": 0.9264, "en_acc": 0.9721, "auroc": 0.9788, "en_auroc": 0.9884, "yes": 0.4036, "en_yes": 0.3528, "gold_yes": 0.3503}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 394

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Vietnamese · prompt_injections · acc

Solar Decide Flash, same named hosted route; Vietnamese; slice prompt_injections. {"answered": 1968, "cases": 1968, "complete": true}

Source observations: {"n": 115, "acc": 0.7565, "en_acc": 0.8, "auroc": 0.8514, "en_auroc": 0.9053, "yes": 0.4609, "en_yes": 0.4, "gold_yes": 0.513}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 115

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Vietnamese · prompt_injections · auroc

Solar Decide Flash, same named hosted route; Vietnamese; slice prompt_injections. {"answered": 1968, "cases": 1968, "complete": true}

Source observations: {"n": 115, "acc": 0.7565, "en_acc": 0.8, "auroc": 0.8514, "en_auroc": 0.9053, "yes": 0.4609, "en_yes": 0.4, "gold_yes": 0.513}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 115

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Vietnamese · prompt_injections · en_acc

Solar Decide Flash, same named hosted route; Vietnamese; slice prompt_injections. {"answered": 1968, "cases": 1968, "complete": true}

Source observations: {"n": 115, "acc": 0.7565, "en_acc": 0.8, "auroc": 0.8514, "en_auroc": 0.9053, "yes": 0.4609, "en_yes": 0.4, "gold_yes": 0.513}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 115

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Vietnamese · prompt_injections · en_auroc

Solar Decide Flash, same named hosted route; Vietnamese; slice prompt_injections. {"answered": 1968, "cases": 1968, "complete": true}

Source observations: {"n": 115, "acc": 0.7565, "en_acc": 0.8, "auroc": 0.8514, "en_auroc": 0.9053, "yes": 0.4609, "en_yes": 0.4, "gold_yes": 0.513}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 115

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Vietnamese · prompt_injections · yes

Solar Decide Flash, same named hosted route; Vietnamese; slice prompt_injections. {"answered": 1968, "cases": 1968, "complete": true}

Source observations: {"n": 115, "acc": 0.7565, "en_acc": 0.8, "auroc": 0.8514, "en_auroc": 0.9053, "yes": 0.4609, "en_yes": 0.4, "gold_yes": 0.513}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 115

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Vietnamese · prompt_injections · en_yes

Solar Decide Flash, same named hosted route; Vietnamese; slice prompt_injections. {"answered": 1968, "cases": 1968, "complete": true}

Source observations: {"n": 115, "acc": 0.7565, "en_acc": 0.8, "auroc": 0.8514, "en_auroc": 0.9053, "yes": 0.4609, "en_yes": 0.4, "gold_yes": 0.513}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 115

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Swahili · overall · acc

Solar Decide Flash, same named hosted route; Swahili; slice overall. {"answered": 1997, "cases": 1997, "complete": true}

Source observations: {"n": 1997, "acc": 0.5541, "en_acc": 0.7445, "delta": -0.1904, "ci": [-0.2197, -0.163], "significant": true, "auroc": 0.5718, "en_auroc": 0.8471}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 1997

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Swahili · overall · auroc

Solar Decide Flash, same named hosted route; Swahili; slice overall. {"answered": 1997, "cases": 1997, "complete": true}

Source observations: {"n": 1997, "acc": 0.5541, "en_acc": 0.7445, "delta": -0.1904, "ci": [-0.2197, -0.163], "significant": true, "auroc": 0.5718, "en_auroc": 0.8471}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 1997

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Swahili · overall · en_acc

Solar Decide Flash, same named hosted route; Swahili; slice overall. {"answered": 1997, "cases": 1997, "complete": true}

Source observations: {"n": 1997, "acc": 0.5541, "en_acc": 0.7445, "delta": -0.1904, "ci": [-0.2197, -0.163], "significant": true, "auroc": 0.5718, "en_auroc": 0.8471}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 1997

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Swahili · overall · en_auroc

Solar Decide Flash, same named hosted route; Swahili; slice overall. {"answered": 1997, "cases": 1997, "complete": true}

Source observations: {"n": 1997, "acc": 0.5541, "en_acc": 0.7445, "delta": -0.1904, "ci": [-0.2197, -0.163], "significant": true, "auroc": 0.5718, "en_auroc": 0.8471}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 1997

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Swahili · overall · delta

Solar Decide Flash, same named hosted route; Swahili; slice overall. {"answered": 1997, "cases": 1997, "complete": true}

Source observations: {"n": 1997, "acc": 0.5541, "en_acc": 0.7445, "delta": -0.1904, "ci": [-0.2197, -0.163], "significant": true, "auroc": 0.5718, "en_auroc": 0.8471}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 1997

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Swahili · aegis2 · acc

Solar Decide Flash, same named hosted route; Swahili; slice aegis2. {"answered": 1997, "cases": 1997, "complete": true}

Source observations: {"n": 500, "acc": 0.5, "en_acc": 0.736, "auroc": 0.5252, "en_auroc": 0.8451, "yes": 0.0, "en_yes": 0.3, "gold_yes": 0.5}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 500

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Swahili · aegis2 · auroc

Solar Decide Flash, same named hosted route; Swahili; slice aegis2. {"answered": 1997, "cases": 1997, "complete": true}

Source observations: {"n": 500, "acc": 0.5, "en_acc": 0.736, "auroc": 0.5252, "en_auroc": 0.8451, "yes": 0.0, "en_yes": 0.3, "gold_yes": 0.5}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 500

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Swahili · aegis2 · en_acc

Solar Decide Flash, same named hosted route; Swahili; slice aegis2. {"answered": 1997, "cases": 1997, "complete": true}

Source observations: {"n": 500, "acc": 0.5, "en_acc": 0.736, "auroc": 0.5252, "en_auroc": 0.8451, "yes": 0.0, "en_yes": 0.3, "gold_yes": 0.5}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 500

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Swahili · aegis2 · en_auroc

Solar Decide Flash, same named hosted route; Swahili; slice aegis2. {"answered": 1997, "cases": 1997, "complete": true}

Source observations: {"n": 500, "acc": 0.5, "en_acc": 0.736, "auroc": 0.5252, "en_auroc": 0.8451, "yes": 0.0, "en_yes": 0.3, "gold_yes": 0.5}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 500

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Swahili · aegis2 · yes

Solar Decide Flash, same named hosted route; Swahili; slice aegis2. {"answered": 1997, "cases": 1997, "complete": true}

Source observations: {"n": 500, "acc": 0.5, "en_acc": 0.736, "auroc": 0.5252, "en_auroc": 0.8451, "yes": 0.0, "en_yes": 0.3, "gold_yes": 0.5}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 500

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Swahili · aegis2 · en_yes

Solar Decide Flash, same named hosted route; Swahili; slice aegis2. {"answered": 1997, "cases": 1997, "complete": true}

Source observations: {"n": 500, "acc": 0.5, "en_acc": 0.736, "auroc": 0.5252, "en_auroc": 0.8451, "yes": 0.0, "en_yes": 0.3, "gold_yes": 0.5}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 500

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Swahili · aegis2_response · acc

Solar Decide Flash, same named hosted route; Swahili; slice aegis2_response. {"answered": 1997, "cases": 1997, "complete": true}

Source observations: {"n": 499, "acc": 0.501, "en_acc": 0.7194, "auroc": 0.3607, "en_auroc": 0.8642, "yes": 0.0, "en_yes": 0.2786, "gold_yes": 0.499}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 499

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Swahili · aegis2_response · auroc

Solar Decide Flash, same named hosted route; Swahili; slice aegis2_response. {"answered": 1997, "cases": 1997, "complete": true}

Source observations: {"n": 499, "acc": 0.501, "en_acc": 0.7194, "auroc": 0.3607, "en_auroc": 0.8642, "yes": 0.0, "en_yes": 0.2786, "gold_yes": 0.499}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 499

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Swahili · aegis2_response · en_acc

Solar Decide Flash, same named hosted route; Swahili; slice aegis2_response. {"answered": 1997, "cases": 1997, "complete": true}

Source observations: {"n": 499, "acc": 0.501, "en_acc": 0.7194, "auroc": 0.3607, "en_auroc": 0.8642, "yes": 0.0, "en_yes": 0.2786, "gold_yes": 0.499}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 499

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Swahili · aegis2_response · en_auroc

Solar Decide Flash, same named hosted route; Swahili; slice aegis2_response. {"answered": 1997, "cases": 1997, "complete": true}

Source observations: {"n": 499, "acc": 0.501, "en_acc": 0.7194, "auroc": 0.3607, "en_auroc": 0.8642, "yes": 0.0, "en_yes": 0.2786, "gold_yes": 0.499}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 499

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Swahili · aegis2_response · yes

Solar Decide Flash, same named hosted route; Swahili; slice aegis2_response. {"answered": 1997, "cases": 1997, "complete": true}

Source observations: {"n": 499, "acc": 0.501, "en_acc": 0.7194, "auroc": 0.3607, "en_auroc": 0.8642, "yes": 0.0, "en_yes": 0.2786, "gold_yes": 0.499}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 499

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Swahili · aegis2_response · en_yes

Solar Decide Flash, same named hosted route; Swahili; slice aegis2_response. {"answered": 1997, "cases": 1997, "complete": true}

Source observations: {"n": 499, "acc": 0.501, "en_acc": 0.7194, "auroc": 0.3607, "en_auroc": 0.8642, "yes": 0.0, "en_yes": 0.2786, "gold_yes": 0.499}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 499

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Swahili · atbench500 · acc

Solar Decide Flash, same named hosted route; Swahili; slice atbench500. {"answered": 1997, "cases": 1997, "complete": true}

Source observations: {"n": 484, "acc": 0.4876, "en_acc": 0.5041, "auroc": 0.5194, "en_auroc": 0.6327, "yes": 0.2087, "en_yes": 0.8616, "gold_yes": 0.5021}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 484

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Swahili · atbench500 · auroc

Solar Decide Flash, same named hosted route; Swahili; slice atbench500. {"answered": 1997, "cases": 1997, "complete": true}

Source observations: {"n": 484, "acc": 0.4876, "en_acc": 0.5041, "auroc": 0.5194, "en_auroc": 0.6327, "yes": 0.2087, "en_yes": 0.8616, "gold_yes": 0.5021}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 484

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Swahili · atbench500 · en_acc

Solar Decide Flash, same named hosted route; Swahili; slice atbench500. {"answered": 1997, "cases": 1997, "complete": true}

Source observations: {"n": 484, "acc": 0.4876, "en_acc": 0.5041, "auroc": 0.5194, "en_auroc": 0.6327, "yes": 0.2087, "en_yes": 0.8616, "gold_yes": 0.5021}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 484

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Swahili · atbench500 · en_auroc

Solar Decide Flash, same named hosted route; Swahili; slice atbench500. {"answered": 1997, "cases": 1997, "complete": true}

Source observations: {"n": 484, "acc": 0.4876, "en_acc": 0.5041, "auroc": 0.5194, "en_auroc": 0.6327, "yes": 0.2087, "en_yes": 0.8616, "gold_yes": 0.5021}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 484

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Swahili · atbench500 · yes

Solar Decide Flash, same named hosted route; Swahili; slice atbench500. {"answered": 1997, "cases": 1997, "complete": true}

Source observations: {"n": 484, "acc": 0.4876, "en_acc": 0.5041, "auroc": 0.5194, "en_auroc": 0.6327, "yes": 0.2087, "en_yes": 0.8616, "gold_yes": 0.5021}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 484

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Swahili · atbench500 · en_yes

Solar Decide Flash, same named hosted route; Swahili; slice atbench500. {"answered": 1997, "cases": 1997, "complete": true}

Source observations: {"n": 484, "acc": 0.4876, "en_acc": 0.5041, "auroc": 0.5194, "en_auroc": 0.6327, "yes": 0.2087, "en_yes": 0.8616, "gold_yes": 0.5021}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 484

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Swahili · jailbreak_classification · acc

Solar Decide Flash, same named hosted route; Swahili; slice jailbreak_classification. {"answered": 1997, "cases": 1997, "complete": true}

Source observations: {"n": 398, "acc": 0.799, "en_acc": 0.9698, "auroc": 0.8394, "en_auroc": 0.9882, "yes": 0.1633, "en_yes": 0.3492, "gold_yes": 0.3492}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 398

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Swahili · jailbreak_classification · auroc

Solar Decide Flash, same named hosted route; Swahili; slice jailbreak_classification. {"answered": 1997, "cases": 1997, "complete": true}

Source observations: {"n": 398, "acc": 0.799, "en_acc": 0.9698, "auroc": 0.8394, "en_auroc": 0.9882, "yes": 0.1633, "en_yes": 0.3492, "gold_yes": 0.3492}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 398

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Swahili · jailbreak_classification · en_acc

Solar Decide Flash, same named hosted route; Swahili; slice jailbreak_classification. {"answered": 1997, "cases": 1997, "complete": true}

Source observations: {"n": 398, "acc": 0.799, "en_acc": 0.9698, "auroc": 0.8394, "en_auroc": 0.9882, "yes": 0.1633, "en_yes": 0.3492, "gold_yes": 0.3492}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 398

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Swahili · jailbreak_classification · en_auroc

Solar Decide Flash, same named hosted route; Swahili; slice jailbreak_classification. {"answered": 1997, "cases": 1997, "complete": true}

Source observations: {"n": 398, "acc": 0.799, "en_acc": 0.9698, "auroc": 0.8394, "en_auroc": 0.9882, "yes": 0.1633, "en_yes": 0.3492, "gold_yes": 0.3492}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 398

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Swahili · jailbreak_classification · yes

Solar Decide Flash, same named hosted route; Swahili; slice jailbreak_classification. {"answered": 1997, "cases": 1997, "complete": true}

Source observations: {"n": 398, "acc": 0.799, "en_acc": 0.9698, "auroc": 0.8394, "en_auroc": 0.9882, "yes": 0.1633, "en_yes": 0.3492, "gold_yes": 0.3492}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 398

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Swahili · jailbreak_classification · en_yes

Solar Decide Flash, same named hosted route; Swahili; slice jailbreak_classification. {"answered": 1997, "cases": 1997, "complete": true}

Source observations: {"n": 398, "acc": 0.799, "en_acc": 0.9698, "auroc": 0.8394, "en_auroc": 0.9882, "yes": 0.1633, "en_yes": 0.3492, "gold_yes": 0.3492}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 398

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Swahili · prompt_injections · acc

Solar Decide Flash, same named hosted route; Swahili; slice prompt_injections. {"answered": 1997, "cases": 1997, "complete": true}

Source observations: {"n": 116, "acc": 0.4828, "en_acc": 0.7931, "auroc": 0.6141, "en_auroc": 0.9051, "yes": 0.0, "en_yes": 0.3966, "gold_yes": 0.5172}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 116

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Swahili · prompt_injections · auroc

Solar Decide Flash, same named hosted route; Swahili; slice prompt_injections. {"answered": 1997, "cases": 1997, "complete": true}

Source observations: {"n": 116, "acc": 0.4828, "en_acc": 0.7931, "auroc": 0.6141, "en_auroc": 0.9051, "yes": 0.0, "en_yes": 0.3966, "gold_yes": 0.5172}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 116

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Swahili · prompt_injections · en_acc

Solar Decide Flash, same named hosted route; Swahili; slice prompt_injections. {"answered": 1997, "cases": 1997, "complete": true}

Source observations: {"n": 116, "acc": 0.4828, "en_acc": 0.7931, "auroc": 0.6141, "en_auroc": 0.9051, "yes": 0.0, "en_yes": 0.3966, "gold_yes": 0.5172}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 116

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Swahili · prompt_injections · en_auroc

Solar Decide Flash, same named hosted route; Swahili; slice prompt_injections. {"answered": 1997, "cases": 1997, "complete": true}

Source observations: {"n": 116, "acc": 0.4828, "en_acc": 0.7931, "auroc": 0.6141, "en_auroc": 0.9051, "yes": 0.0, "en_yes": 0.3966, "gold_yes": 0.5172}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 116

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Swahili · prompt_injections · yes

Solar Decide Flash, same named hosted route; Swahili; slice prompt_injections. {"answered": 1997, "cases": 1997, "complete": true}

Source observations: {"n": 116, "acc": 0.4828, "en_acc": 0.7931, "auroc": 0.6141, "en_auroc": 0.9051, "yes": 0.0, "en_yes": 0.3966, "gold_yes": 0.5172}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 116

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Swahili · prompt_injections · en_yes

Solar Decide Flash, same named hosted route; Swahili; slice prompt_injections. {"answered": 1997, "cases": 1997, "complete": true}

Source observations: {"n": 116, "acc": 0.4828, "en_acc": 0.7931, "auroc": 0.6141, "en_auroc": 0.9051, "yes": 0.0, "en_yes": 0.3966, "gold_yes": 0.5172}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 116

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Spanish · overall · acc

Solar Decide Flash, same named hosted route; Spanish; slice overall. {"answered": 1959, "cases": 1959, "complete": true}

Source observations: {"n": 1959, "acc": 0.7213, "en_acc": 0.7446, "delta": -0.0233, "ci": [-0.0426, -0.0042], "significant": true, "auroc": 0.8295, "en_auroc": 0.8488}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 1959

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Spanish · overall · auroc

Solar Decide Flash, same named hosted route; Spanish; slice overall. {"answered": 1959, "cases": 1959, "complete": true}

Source observations: {"n": 1959, "acc": 0.7213, "en_acc": 0.7446, "delta": -0.0233, "ci": [-0.0426, -0.0042], "significant": true, "auroc": 0.8295, "en_auroc": 0.8488}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 1959

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Spanish · overall · en_acc

Solar Decide Flash, same named hosted route; Spanish; slice overall. {"answered": 1959, "cases": 1959, "complete": true}

Source observations: {"n": 1959, "acc": 0.7213, "en_acc": 0.7446, "delta": -0.0233, "ci": [-0.0426, -0.0042], "significant": true, "auroc": 0.8295, "en_auroc": 0.8488}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 1959

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Spanish · overall · en_auroc

Solar Decide Flash, same named hosted route; Spanish; slice overall. {"answered": 1959, "cases": 1959, "complete": true}

Source observations: {"n": 1959, "acc": 0.7213, "en_acc": 0.7446, "delta": -0.0233, "ci": [-0.0426, -0.0042], "significant": true, "auroc": 0.8295, "en_auroc": 0.8488}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 1959

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Spanish · overall · delta

Solar Decide Flash, same named hosted route; Spanish; slice overall. {"answered": 1959, "cases": 1959, "complete": true}

Source observations: {"n": 1959, "acc": 0.7213, "en_acc": 0.7446, "delta": -0.0233, "ci": [-0.0426, -0.0042], "significant": true, "auroc": 0.8295, "en_auroc": 0.8488}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 1959

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Spanish · aegis2 · acc

Solar Decide Flash, same named hosted route; Spanish; slice aegis2. {"answered": 1959, "cases": 1959, "complete": true}

Source observations: {"n": 499, "acc": 0.7014, "en_acc": 0.7355, "auroc": 0.8311, "en_auroc": 0.8461, "yes": 0.2265, "en_yes": 0.3006, "gold_yes": 0.501}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 499

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Spanish · aegis2 · auroc

Solar Decide Flash, same named hosted route; Spanish; slice aegis2. {"answered": 1959, "cases": 1959, "complete": true}

Source observations: {"n": 499, "acc": 0.7014, "en_acc": 0.7355, "auroc": 0.8311, "en_auroc": 0.8461, "yes": 0.2265, "en_yes": 0.3006, "gold_yes": 0.501}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 499

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Spanish · aegis2 · en_acc

Solar Decide Flash, same named hosted route; Spanish; slice aegis2. {"answered": 1959, "cases": 1959, "complete": true}

Source observations: {"n": 499, "acc": 0.7014, "en_acc": 0.7355, "auroc": 0.8311, "en_auroc": 0.8461, "yes": 0.2265, "en_yes": 0.3006, "gold_yes": 0.501}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 499

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Spanish · aegis2 · en_auroc

Solar Decide Flash, same named hosted route; Spanish; slice aegis2. {"answered": 1959, "cases": 1959, "complete": true}

Source observations: {"n": 499, "acc": 0.7014, "en_acc": 0.7355, "auroc": 0.8311, "en_auroc": 0.8461, "yes": 0.2265, "en_yes": 0.3006, "gold_yes": 0.501}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 499

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Spanish · aegis2 · yes

Solar Decide Flash, same named hosted route; Spanish; slice aegis2. {"answered": 1959, "cases": 1959, "complete": true}

Source observations: {"n": 499, "acc": 0.7014, "en_acc": 0.7355, "auroc": 0.8311, "en_auroc": 0.8461, "yes": 0.2265, "en_yes": 0.3006, "gold_yes": 0.501}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 499

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Spanish · aegis2 · en_yes

Solar Decide Flash, same named hosted route; Spanish; slice aegis2. {"answered": 1959, "cases": 1959, "complete": true}

Source observations: {"n": 499, "acc": 0.7014, "en_acc": 0.7355, "auroc": 0.8311, "en_auroc": 0.8461, "yes": 0.2265, "en_yes": 0.3006, "gold_yes": 0.501}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 499

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Spanish · aegis2_response · acc

Solar Decide Flash, same named hosted route; Spanish; slice aegis2_response. {"answered": 1959, "cases": 1959, "complete": true}

Source observations: {"n": 500, "acc": 0.73, "en_acc": 0.72, "auroc": 0.851, "en_auroc": 0.8647, "yes": 0.314, "en_yes": 0.28, "gold_yes": 0.5}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 500

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Spanish · aegis2_response · auroc

Solar Decide Flash, same named hosted route; Spanish; slice aegis2_response. {"answered": 1959, "cases": 1959, "complete": true}

Source observations: {"n": 500, "acc": 0.73, "en_acc": 0.72, "auroc": 0.851, "en_auroc": 0.8647, "yes": 0.314, "en_yes": 0.28, "gold_yes": 0.5}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 500

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Spanish · aegis2_response · en_acc

Solar Decide Flash, same named hosted route; Spanish; slice aegis2_response. {"answered": 1959, "cases": 1959, "complete": true}

Source observations: {"n": 500, "acc": 0.73, "en_acc": 0.72, "auroc": 0.851, "en_auroc": 0.8647, "yes": 0.314, "en_yes": 0.28, "gold_yes": 0.5}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 500

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Spanish · aegis2_response · en_auroc

Solar Decide Flash, same named hosted route; Spanish; slice aegis2_response. {"answered": 1959, "cases": 1959, "complete": true}

Source observations: {"n": 500, "acc": 0.73, "en_acc": 0.72, "auroc": 0.851, "en_auroc": 0.8647, "yes": 0.314, "en_yes": 0.28, "gold_yes": 0.5}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 500

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Spanish · aegis2_response · yes

Solar Decide Flash, same named hosted route; Spanish; slice aegis2_response. {"answered": 1959, "cases": 1959, "complete": true}

Source observations: {"n": 500, "acc": 0.73, "en_acc": 0.72, "auroc": 0.851, "en_auroc": 0.8647, "yes": 0.314, "en_yes": 0.28, "gold_yes": 0.5}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 500

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Spanish · aegis2_response · en_yes

Solar Decide Flash, same named hosted route; Spanish; slice aegis2_response. {"answered": 1959, "cases": 1959, "complete": true}

Source observations: {"n": 500, "acc": 0.73, "en_acc": 0.72, "auroc": 0.851, "en_auroc": 0.8647, "yes": 0.314, "en_yes": 0.28, "gold_yes": 0.5}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 500

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Spanish · atbench500 · acc

Solar Decide Flash, same named hosted route; Spanish; slice atbench500. {"answered": 1959, "cases": 1959, "complete": true}

Source observations: {"n": 451, "acc": 0.4834, "en_acc": 0.49, "auroc": 0.5666, "en_auroc": 0.6396, "yes": 0.9756, "en_yes": 0.867, "gold_yes": 0.4723}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 451

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Spanish · atbench500 · auroc

Solar Decide Flash, same named hosted route; Spanish; slice atbench500. {"answered": 1959, "cases": 1959, "complete": true}

Source observations: {"n": 451, "acc": 0.4834, "en_acc": 0.49, "auroc": 0.5666, "en_auroc": 0.6396, "yes": 0.9756, "en_yes": 0.867, "gold_yes": 0.4723}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 451

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Spanish · atbench500 · en_acc

Solar Decide Flash, same named hosted route; Spanish; slice atbench500. {"answered": 1959, "cases": 1959, "complete": true}

Source observations: {"n": 451, "acc": 0.4834, "en_acc": 0.49, "auroc": 0.5666, "en_auroc": 0.6396, "yes": 0.9756, "en_yes": 0.867, "gold_yes": 0.4723}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 451

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Spanish · atbench500 · en_auroc

Solar Decide Flash, same named hosted route; Spanish; slice atbench500. {"answered": 1959, "cases": 1959, "complete": true}

Source observations: {"n": 451, "acc": 0.4834, "en_acc": 0.49, "auroc": 0.5666, "en_auroc": 0.6396, "yes": 0.9756, "en_yes": 0.867, "gold_yes": 0.4723}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 451

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Spanish · atbench500 · yes

Solar Decide Flash, same named hosted route; Spanish; slice atbench500. {"answered": 1959, "cases": 1959, "complete": true}

Source observations: {"n": 451, "acc": 0.4834, "en_acc": 0.49, "auroc": 0.5666, "en_auroc": 0.6396, "yes": 0.9756, "en_yes": 0.867, "gold_yes": 0.4723}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 451

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Spanish · atbench500 · en_yes

Solar Decide Flash, same named hosted route; Spanish; slice atbench500. {"answered": 1959, "cases": 1959, "complete": true}

Source observations: {"n": 451, "acc": 0.4834, "en_acc": 0.49, "auroc": 0.5666, "en_auroc": 0.6396, "yes": 0.9756, "en_yes": 0.867, "gold_yes": 0.4723}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 451

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Spanish · jailbreak_classification · acc

Solar Decide Flash, same named hosted route; Spanish; slice jailbreak_classification. {"answered": 1959, "cases": 1959, "complete": true}

Source observations: {"n": 396, "acc": 0.9747, "en_acc": 0.9722, "auroc": 0.9861, "en_auroc": 0.9883, "yes": 0.3384, "en_yes": 0.351, "gold_yes": 0.3485}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 396

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Spanish · jailbreak_classification · auroc

Solar Decide Flash, same named hosted route; Spanish; slice jailbreak_classification. {"answered": 1959, "cases": 1959, "complete": true}

Source observations: {"n": 396, "acc": 0.9747, "en_acc": 0.9722, "auroc": 0.9861, "en_auroc": 0.9883, "yes": 0.3384, "en_yes": 0.351, "gold_yes": 0.3485}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 396

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Spanish · jailbreak_classification · en_acc

Solar Decide Flash, same named hosted route; Spanish; slice jailbreak_classification. {"answered": 1959, "cases": 1959, "complete": true}

Source observations: {"n": 396, "acc": 0.9747, "en_acc": 0.9722, "auroc": 0.9861, "en_auroc": 0.9883, "yes": 0.3384, "en_yes": 0.351, "gold_yes": 0.3485}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 396

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Spanish · jailbreak_classification · en_auroc

Solar Decide Flash, same named hosted route; Spanish; slice jailbreak_classification. {"answered": 1959, "cases": 1959, "complete": true}

Source observations: {"n": 396, "acc": 0.9747, "en_acc": 0.9722, "auroc": 0.9861, "en_auroc": 0.9883, "yes": 0.3384, "en_yes": 0.351, "gold_yes": 0.3485}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 396

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Spanish · jailbreak_classification · yes

Solar Decide Flash, same named hosted route; Spanish; slice jailbreak_classification. {"answered": 1959, "cases": 1959, "complete": true}

Source observations: {"n": 396, "acc": 0.9747, "en_acc": 0.9722, "auroc": 0.9861, "en_auroc": 0.9883, "yes": 0.3384, "en_yes": 0.351, "gold_yes": 0.3485}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 396

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Spanish · jailbreak_classification · en_yes

Solar Decide Flash, same named hosted route; Spanish; slice jailbreak_classification. {"answered": 1959, "cases": 1959, "complete": true}

Source observations: {"n": 396, "acc": 0.9747, "en_acc": 0.9722, "auroc": 0.9861, "en_auroc": 0.9883, "yes": 0.3384, "en_yes": 0.351, "gold_yes": 0.3485}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 396

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Spanish · prompt_injections · acc

Solar Decide Flash, same named hosted route; Spanish; slice prompt_injections. {"answered": 1959, "cases": 1959, "complete": true}

Source observations: {"n": 113, "acc": 0.7168, "en_acc": 0.8053, "auroc": 0.9129, "en_auroc": 0.9051, "yes": 0.2389, "en_yes": 0.3982, "gold_yes": 0.5044}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 113

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Spanish · prompt_injections · auroc

Solar Decide Flash, same named hosted route; Spanish; slice prompt_injections. {"answered": 1959, "cases": 1959, "complete": true}

Source observations: {"n": 113, "acc": 0.7168, "en_acc": 0.8053, "auroc": 0.9129, "en_auroc": 0.9051, "yes": 0.2389, "en_yes": 0.3982, "gold_yes": 0.5044}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 113

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Spanish · prompt_injections · en_acc

Solar Decide Flash, same named hosted route; Spanish; slice prompt_injections. {"answered": 1959, "cases": 1959, "complete": true}

Source observations: {"n": 113, "acc": 0.7168, "en_acc": 0.8053, "auroc": 0.9129, "en_auroc": 0.9051, "yes": 0.2389, "en_yes": 0.3982, "gold_yes": 0.5044}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 113

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Spanish · prompt_injections · en_auroc

Solar Decide Flash, same named hosted route; Spanish; slice prompt_injections. {"answered": 1959, "cases": 1959, "complete": true}

Source observations: {"n": 113, "acc": 0.7168, "en_acc": 0.8053, "auroc": 0.9129, "en_auroc": 0.9051, "yes": 0.2389, "en_yes": 0.3982, "gold_yes": 0.5044}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 113

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Spanish · prompt_injections · yes

Solar Decide Flash, same named hosted route; Spanish; slice prompt_injections. {"answered": 1959, "cases": 1959, "complete": true}

Source observations: {"n": 113, "acc": 0.7168, "en_acc": 0.8053, "auroc": 0.9129, "en_auroc": 0.9051, "yes": 0.2389, "en_yes": 0.3982, "gold_yes": 0.5044}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 113

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Spanish · prompt_injections · en_yes

Solar Decide Flash, same named hosted route; Spanish; slice prompt_injections. {"answered": 1959, "cases": 1959, "complete": true}

Source observations: {"n": 113, "acc": 0.7168, "en_acc": 0.8053, "auroc": 0.9129, "en_auroc": 0.9051, "yes": 0.2389, "en_yes": 0.3982, "gold_yes": 0.5044}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 113

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · French · overall · acc

Solar Decide Flash, same named hosted route; French; slice overall. {"answered": 1969, "cases": 1969, "complete": true}

Source observations: {"n": 1969, "acc": 0.7256, "en_acc": 0.7489, "delta": -0.0233, "ci": [-0.0432, -0.0055], "significant": true, "auroc": 0.8205, "en_auroc": 0.8475}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 1969

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · French · overall · auroc

Solar Decide Flash, same named hosted route; French; slice overall. {"answered": 1969, "cases": 1969, "complete": true}

Source observations: {"n": 1969, "acc": 0.7256, "en_acc": 0.7489, "delta": -0.0233, "ci": [-0.0432, -0.0055], "significant": true, "auroc": 0.8205, "en_auroc": 0.8475}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 1969

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · French · overall · en_acc

Solar Decide Flash, same named hosted route; French; slice overall. {"answered": 1969, "cases": 1969, "complete": true}

Source observations: {"n": 1969, "acc": 0.7256, "en_acc": 0.7489, "delta": -0.0233, "ci": [-0.0432, -0.0055], "significant": true, "auroc": 0.8205, "en_auroc": 0.8475}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 1969

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · French · overall · en_auroc

Solar Decide Flash, same named hosted route; French; slice overall. {"answered": 1969, "cases": 1969, "complete": true}

Source observations: {"n": 1969, "acc": 0.7256, "en_acc": 0.7489, "delta": -0.0233, "ci": [-0.0432, -0.0055], "significant": true, "auroc": 0.8205, "en_auroc": 0.8475}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 1969

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · French · overall · delta

Solar Decide Flash, same named hosted route; French; slice overall. {"answered": 1969, "cases": 1969, "complete": true}

Source observations: {"n": 1969, "acc": 0.7256, "en_acc": 0.7489, "delta": -0.0233, "ci": [-0.0432, -0.0055], "significant": true, "auroc": 0.8205, "en_auroc": 0.8475}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 1969

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · French · aegis2 · acc

Solar Decide Flash, same named hosted route; French; slice aegis2. {"answered": 1969, "cases": 1969, "complete": true}

Source observations: {"n": 500, "acc": 0.702, "en_acc": 0.736, "auroc": 0.8363, "en_auroc": 0.8451, "yes": 0.218, "en_yes": 0.3, "gold_yes": 0.5}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 500

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · French · aegis2 · auroc

Solar Decide Flash, same named hosted route; French; slice aegis2. {"answered": 1969, "cases": 1969, "complete": true}

Source observations: {"n": 500, "acc": 0.702, "en_acc": 0.736, "auroc": 0.8363, "en_auroc": 0.8451, "yes": 0.218, "en_yes": 0.3, "gold_yes": 0.5}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 500

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · French · aegis2 · en_acc

Solar Decide Flash, same named hosted route; French; slice aegis2. {"answered": 1969, "cases": 1969, "complete": true}

Source observations: {"n": 500, "acc": 0.702, "en_acc": 0.736, "auroc": 0.8363, "en_auroc": 0.8451, "yes": 0.218, "en_yes": 0.3, "gold_yes": 0.5}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 500

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · French · aegis2 · en_auroc

Solar Decide Flash, same named hosted route; French; slice aegis2. {"answered": 1969, "cases": 1969, "complete": true}

Source observations: {"n": 500, "acc": 0.702, "en_acc": 0.736, "auroc": 0.8363, "en_auroc": 0.8451, "yes": 0.218, "en_yes": 0.3, "gold_yes": 0.5}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 500

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · French · aegis2 · yes

Solar Decide Flash, same named hosted route; French; slice aegis2. {"answered": 1969, "cases": 1969, "complete": true}

Source observations: {"n": 500, "acc": 0.702, "en_acc": 0.736, "auroc": 0.8363, "en_auroc": 0.8451, "yes": 0.218, "en_yes": 0.3, "gold_yes": 0.5}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 500

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · French · aegis2 · en_yes

Solar Decide Flash, same named hosted route; French; slice aegis2. {"answered": 1969, "cases": 1969, "complete": true}

Source observations: {"n": 500, "acc": 0.702, "en_acc": 0.736, "auroc": 0.8363, "en_auroc": 0.8451, "yes": 0.218, "en_yes": 0.3, "gold_yes": 0.5}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 500

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · French · aegis2_response · acc

Solar Decide Flash, same named hosted route; French; slice aegis2_response. {"answered": 1969, "cases": 1969, "complete": true}

Source observations: {"n": 500, "acc": 0.754, "en_acc": 0.72, "auroc": 0.8558, "en_auroc": 0.8647, "yes": 0.37, "en_yes": 0.28, "gold_yes": 0.5}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 500

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · French · aegis2_response · auroc

Solar Decide Flash, same named hosted route; French; slice aegis2_response. {"answered": 1969, "cases": 1969, "complete": true}

Source observations: {"n": 500, "acc": 0.754, "en_acc": 0.72, "auroc": 0.8558, "en_auroc": 0.8647, "yes": 0.37, "en_yes": 0.28, "gold_yes": 0.5}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 500

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · French · aegis2_response · en_acc

Solar Decide Flash, same named hosted route; French; slice aegis2_response. {"answered": 1969, "cases": 1969, "complete": true}

Source observations: {"n": 500, "acc": 0.754, "en_acc": 0.72, "auroc": 0.8558, "en_auroc": 0.8647, "yes": 0.37, "en_yes": 0.28, "gold_yes": 0.5}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 500

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · French · aegis2_response · en_auroc

Solar Decide Flash, same named hosted route; French; slice aegis2_response. {"answered": 1969, "cases": 1969, "complete": true}

Source observations: {"n": 500, "acc": 0.754, "en_acc": 0.72, "auroc": 0.8558, "en_auroc": 0.8647, "yes": 0.37, "en_yes": 0.28, "gold_yes": 0.5}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 500

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · French · aegis2_response · yes

Solar Decide Flash, same named hosted route; French; slice aegis2_response. {"answered": 1969, "cases": 1969, "complete": true}

Source observations: {"n": 500, "acc": 0.754, "en_acc": 0.72, "auroc": 0.8558, "en_auroc": 0.8647, "yes": 0.37, "en_yes": 0.28, "gold_yes": 0.5}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 500

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · French · aegis2_response · en_yes

Solar Decide Flash, same named hosted route; French; slice aegis2_response. {"answered": 1969, "cases": 1969, "complete": true}

Source observations: {"n": 500, "acc": 0.754, "en_acc": 0.72, "auroc": 0.8558, "en_auroc": 0.8647, "yes": 0.37, "en_yes": 0.28, "gold_yes": 0.5}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 500

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · French · atbench500 · acc

Solar Decide Flash, same named hosted route; French; slice atbench500. {"answered": 1969, "cases": 1969, "complete": true}

Source observations: {"n": 459, "acc": 0.5229, "en_acc": 0.5163, "auroc": 0.63, "en_auroc": 0.6344, "yes": 0.817, "en_yes": 0.8584, "gold_yes": 0.5142}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 459

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · French · atbench500 · auroc

Solar Decide Flash, same named hosted route; French; slice atbench500. {"answered": 1969, "cases": 1969, "complete": true}

Source observations: {"n": 459, "acc": 0.5229, "en_acc": 0.5163, "auroc": 0.63, "en_auroc": 0.6344, "yes": 0.817, "en_yes": 0.8584, "gold_yes": 0.5142}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 459

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · French · atbench500 · en_acc

Solar Decide Flash, same named hosted route; French; slice atbench500. {"answered": 1969, "cases": 1969, "complete": true}

Source observations: {"n": 459, "acc": 0.5229, "en_acc": 0.5163, "auroc": 0.63, "en_auroc": 0.6344, "yes": 0.817, "en_yes": 0.8584, "gold_yes": 0.5142}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 459

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · French · atbench500 · en_auroc

Solar Decide Flash, same named hosted route; French; slice atbench500. {"answered": 1969, "cases": 1969, "complete": true}

Source observations: {"n": 459, "acc": 0.5229, "en_acc": 0.5163, "auroc": 0.63, "en_auroc": 0.6344, "yes": 0.817, "en_yes": 0.8584, "gold_yes": 0.5142}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 459

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · French · atbench500 · yes

Solar Decide Flash, same named hosted route; French; slice atbench500. {"answered": 1969, "cases": 1969, "complete": true}

Source observations: {"n": 459, "acc": 0.5229, "en_acc": 0.5163, "auroc": 0.63, "en_auroc": 0.6344, "yes": 0.817, "en_yes": 0.8584, "gold_yes": 0.5142}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 459

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · French · atbench500 · en_yes

Solar Decide Flash, same named hosted route; French; slice atbench500. {"answered": 1969, "cases": 1969, "complete": true}

Source observations: {"n": 459, "acc": 0.5229, "en_acc": 0.5163, "auroc": 0.63, "en_auroc": 0.6344, "yes": 0.817, "en_yes": 0.8584, "gold_yes": 0.5142}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 459

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · French · jailbreak_classification · acc

Solar Decide Flash, same named hosted route; French; slice jailbreak_classification. {"answered": 1969, "cases": 1969, "complete": true}

Source observations: {"n": 395, "acc": 0.962, "en_acc": 0.9722, "auroc": 0.9867, "en_auroc": 0.9883, "yes": 0.3544, "en_yes": 0.3494, "gold_yes": 0.3468}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 395

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · French · jailbreak_classification · auroc

Solar Decide Flash, same named hosted route; French; slice jailbreak_classification. {"answered": 1969, "cases": 1969, "complete": true}

Source observations: {"n": 395, "acc": 0.962, "en_acc": 0.9722, "auroc": 0.9867, "en_auroc": 0.9883, "yes": 0.3544, "en_yes": 0.3494, "gold_yes": 0.3468}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 395

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · French · jailbreak_classification · en_acc

Solar Decide Flash, same named hosted route; French; slice jailbreak_classification. {"answered": 1969, "cases": 1969, "complete": true}

Source observations: {"n": 395, "acc": 0.962, "en_acc": 0.9722, "auroc": 0.9867, "en_auroc": 0.9883, "yes": 0.3544, "en_yes": 0.3494, "gold_yes": 0.3468}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 395

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · French · jailbreak_classification · en_auroc

Solar Decide Flash, same named hosted route; French; slice jailbreak_classification. {"answered": 1969, "cases": 1969, "complete": true}

Source observations: {"n": 395, "acc": 0.962, "en_acc": 0.9722, "auroc": 0.9867, "en_auroc": 0.9883, "yes": 0.3544, "en_yes": 0.3494, "gold_yes": 0.3468}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 395

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · French · jailbreak_classification · yes

Solar Decide Flash, same named hosted route; French; slice jailbreak_classification. {"answered": 1969, "cases": 1969, "complete": true}

Source observations: {"n": 395, "acc": 0.962, "en_acc": 0.9722, "auroc": 0.9867, "en_auroc": 0.9883, "yes": 0.3544, "en_yes": 0.3494, "gold_yes": 0.3468}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 395

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · French · jailbreak_classification · en_yes

Solar Decide Flash, same named hosted route; French; slice jailbreak_classification. {"answered": 1969, "cases": 1969, "complete": true}

Source observations: {"n": 395, "acc": 0.962, "en_acc": 0.9722, "auroc": 0.9867, "en_auroc": 0.9883, "yes": 0.3544, "en_yes": 0.3494, "gold_yes": 0.3468}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 395

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · French · prompt_injections · acc

Solar Decide Flash, same named hosted route; French; slice prompt_injections. {"answered": 1969, "cases": 1969, "complete": true}

Source observations: {"n": 115, "acc": 0.687, "en_acc": 0.8, "auroc": 0.7936, "en_auroc": 0.9053, "yes": 0.2174, "en_yes": 0.4, "gold_yes": 0.513}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 115

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · French · prompt_injections · auroc

Solar Decide Flash, same named hosted route; French; slice prompt_injections. {"answered": 1969, "cases": 1969, "complete": true}

Source observations: {"n": 115, "acc": 0.687, "en_acc": 0.8, "auroc": 0.7936, "en_auroc": 0.9053, "yes": 0.2174, "en_yes": 0.4, "gold_yes": 0.513}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 115

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · French · prompt_injections · en_acc

Solar Decide Flash, same named hosted route; French; slice prompt_injections. {"answered": 1969, "cases": 1969, "complete": true}

Source observations: {"n": 115, "acc": 0.687, "en_acc": 0.8, "auroc": 0.7936, "en_auroc": 0.9053, "yes": 0.2174, "en_yes": 0.4, "gold_yes": 0.513}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 115

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · French · prompt_injections · en_auroc

Solar Decide Flash, same named hosted route; French; slice prompt_injections. {"answered": 1969, "cases": 1969, "complete": true}

Source observations: {"n": 115, "acc": 0.687, "en_acc": 0.8, "auroc": 0.7936, "en_auroc": 0.9053, "yes": 0.2174, "en_yes": 0.4, "gold_yes": 0.513}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 115

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · French · prompt_injections · yes

Solar Decide Flash, same named hosted route; French; slice prompt_injections. {"answered": 1969, "cases": 1969, "complete": true}

Source observations: {"n": 115, "acc": 0.687, "en_acc": 0.8, "auroc": 0.7936, "en_auroc": 0.9053, "yes": 0.2174, "en_yes": 0.4, "gold_yes": 0.513}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 115

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · French · prompt_injections · en_yes

Solar Decide Flash, same named hosted route; French; slice prompt_injections. {"answered": 1969, "cases": 1969, "complete": true}

Source observations: {"n": 115, "acc": 0.687, "en_acc": 0.8, "auroc": 0.7936, "en_auroc": 0.9053, "yes": 0.2174, "en_yes": 0.4, "gold_yes": 0.513}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 115

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Russian · overall · acc

Solar Decide Flash, same named hosted route; Russian; slice overall. {"answered": 2005, "cases": 2005, "complete": true}

Source observations: {"n": 2005, "acc": 0.7376, "en_acc": 0.7446, "delta": -0.007, "ci": [-0.0272, 0.012], "significant": false, "auroc": 0.8101, "en_auroc": 0.8464}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 2005

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Russian · overall · auroc

Solar Decide Flash, same named hosted route; Russian; slice overall. {"answered": 2005, "cases": 2005, "complete": true}

Source observations: {"n": 2005, "acc": 0.7376, "en_acc": 0.7446, "delta": -0.007, "ci": [-0.0272, 0.012], "significant": false, "auroc": 0.8101, "en_auroc": 0.8464}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 2005

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Russian · overall · en_acc

Solar Decide Flash, same named hosted route; Russian; slice overall. {"answered": 2005, "cases": 2005, "complete": true}

Source observations: {"n": 2005, "acc": 0.7376, "en_acc": 0.7446, "delta": -0.007, "ci": [-0.0272, 0.012], "significant": false, "auroc": 0.8101, "en_auroc": 0.8464}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 2005

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Russian · overall · en_auroc

Solar Decide Flash, same named hosted route; Russian; slice overall. {"answered": 2005, "cases": 2005, "complete": true}

Source observations: {"n": 2005, "acc": 0.7376, "en_acc": 0.7446, "delta": -0.007, "ci": [-0.0272, 0.012], "significant": false, "auroc": 0.8101, "en_auroc": 0.8464}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 2005

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Russian · overall · delta

Solar Decide Flash, same named hosted route; Russian; slice overall. {"answered": 2005, "cases": 2005, "complete": true}

Source observations: {"n": 2005, "acc": 0.7376, "en_acc": 0.7446, "delta": -0.007, "ci": [-0.0272, 0.012], "significant": false, "auroc": 0.8101, "en_auroc": 0.8464}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 2005

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Russian · aegis2 · acc

Solar Decide Flash, same named hosted route; Russian; slice aegis2. {"answered": 2005, "cases": 2005, "complete": true}

Source observations: {"n": 500, "acc": 0.706, "en_acc": 0.736, "auroc": 0.8026, "en_auroc": 0.8451, "yes": 0.254, "en_yes": 0.3, "gold_yes": 0.5}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 500

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Russian · aegis2 · auroc

Solar Decide Flash, same named hosted route; Russian; slice aegis2. {"answered": 2005, "cases": 2005, "complete": true}

Source observations: {"n": 500, "acc": 0.706, "en_acc": 0.736, "auroc": 0.8026, "en_auroc": 0.8451, "yes": 0.254, "en_yes": 0.3, "gold_yes": 0.5}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 500

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Russian · aegis2 · en_acc

Solar Decide Flash, same named hosted route; Russian; slice aegis2. {"answered": 2005, "cases": 2005, "complete": true}

Source observations: {"n": 500, "acc": 0.706, "en_acc": 0.736, "auroc": 0.8026, "en_auroc": 0.8451, "yes": 0.254, "en_yes": 0.3, "gold_yes": 0.5}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 500

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Russian · aegis2 · en_auroc

Solar Decide Flash, same named hosted route; Russian; slice aegis2. {"answered": 2005, "cases": 2005, "complete": true}

Source observations: {"n": 500, "acc": 0.706, "en_acc": 0.736, "auroc": 0.8026, "en_auroc": 0.8451, "yes": 0.254, "en_yes": 0.3, "gold_yes": 0.5}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 500

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Russian · aegis2 · yes

Solar Decide Flash, same named hosted route; Russian; slice aegis2. {"answered": 2005, "cases": 2005, "complete": true}

Source observations: {"n": 500, "acc": 0.706, "en_acc": 0.736, "auroc": 0.8026, "en_auroc": 0.8451, "yes": 0.254, "en_yes": 0.3, "gold_yes": 0.5}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 500

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Russian · aegis2 · en_yes

Solar Decide Flash, same named hosted route; Russian; slice aegis2. {"answered": 2005, "cases": 2005, "complete": true}

Source observations: {"n": 500, "acc": 0.706, "en_acc": 0.736, "auroc": 0.8026, "en_auroc": 0.8451, "yes": 0.254, "en_yes": 0.3, "gold_yes": 0.5}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 500

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Russian · aegis2_response · acc

Solar Decide Flash, same named hosted route; Russian; slice aegis2_response. {"answered": 2005, "cases": 2005, "complete": true}

Source observations: {"n": 500, "acc": 0.724, "en_acc": 0.72, "auroc": 0.8227, "en_auroc": 0.8647, "yes": 0.336, "en_yes": 0.28, "gold_yes": 0.5}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 500

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Russian · aegis2_response · auroc

Solar Decide Flash, same named hosted route; Russian; slice aegis2_response. {"answered": 2005, "cases": 2005, "complete": true}

Source observations: {"n": 500, "acc": 0.724, "en_acc": 0.72, "auroc": 0.8227, "en_auroc": 0.8647, "yes": 0.336, "en_yes": 0.28, "gold_yes": 0.5}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 500

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Russian · aegis2_response · en_acc

Solar Decide Flash, same named hosted route; Russian; slice aegis2_response. {"answered": 2005, "cases": 2005, "complete": true}

Source observations: {"n": 500, "acc": 0.724, "en_acc": 0.72, "auroc": 0.8227, "en_auroc": 0.8647, "yes": 0.336, "en_yes": 0.28, "gold_yes": 0.5}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 500

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Russian · aegis2_response · en_auroc

Solar Decide Flash, same named hosted route; Russian; slice aegis2_response. {"answered": 2005, "cases": 2005, "complete": true}

Source observations: {"n": 500, "acc": 0.724, "en_acc": 0.72, "auroc": 0.8227, "en_auroc": 0.8647, "yes": 0.336, "en_yes": 0.28, "gold_yes": 0.5}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 500

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Russian · aegis2_response · yes

Solar Decide Flash, same named hosted route; Russian; slice aegis2_response. {"answered": 2005, "cases": 2005, "complete": true}

Source observations: {"n": 500, "acc": 0.724, "en_acc": 0.72, "auroc": 0.8227, "en_auroc": 0.8647, "yes": 0.336, "en_yes": 0.28, "gold_yes": 0.5}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 500

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Russian · aegis2_response · en_yes

Solar Decide Flash, same named hosted route; Russian; slice aegis2_response. {"answered": 2005, "cases": 2005, "complete": true}

Source observations: {"n": 500, "acc": 0.724, "en_acc": 0.72, "auroc": 0.8227, "en_auroc": 0.8647, "yes": 0.336, "en_yes": 0.28, "gold_yes": 0.5}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 500

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Russian · atbench500 · acc

Solar Decide Flash, same named hosted route; Russian; slice atbench500. {"answered": 2005, "cases": 2005, "complete": true}

Source observations: {"n": 490, "acc": 0.498, "en_acc": 0.5041, "auroc": 0.5349, "en_auroc": 0.6287, "yes": 0.9163, "en_yes": 0.8612, "gold_yes": 0.5041}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 490

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Russian · atbench500 · auroc

Solar Decide Flash, same named hosted route; Russian; slice atbench500. {"answered": 2005, "cases": 2005, "complete": true}

Source observations: {"n": 490, "acc": 0.498, "en_acc": 0.5041, "auroc": 0.5349, "en_auroc": 0.6287, "yes": 0.9163, "en_yes": 0.8612, "gold_yes": 0.5041}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 490

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Russian · atbench500 · en_acc

Solar Decide Flash, same named hosted route; Russian; slice atbench500. {"answered": 2005, "cases": 2005, "complete": true}

Source observations: {"n": 490, "acc": 0.498, "en_acc": 0.5041, "auroc": 0.5349, "en_auroc": 0.6287, "yes": 0.9163, "en_yes": 0.8612, "gold_yes": 0.5041}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 490

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Russian · atbench500 · en_auroc

Solar Decide Flash, same named hosted route; Russian; slice atbench500. {"answered": 2005, "cases": 2005, "complete": true}

Source observations: {"n": 490, "acc": 0.498, "en_acc": 0.5041, "auroc": 0.5349, "en_auroc": 0.6287, "yes": 0.9163, "en_yes": 0.8612, "gold_yes": 0.5041}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 490

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Russian · atbench500 · yes

Solar Decide Flash, same named hosted route; Russian; slice atbench500. {"answered": 2005, "cases": 2005, "complete": true}

Source observations: {"n": 490, "acc": 0.498, "en_acc": 0.5041, "auroc": 0.5349, "en_auroc": 0.6287, "yes": 0.9163, "en_yes": 0.8612, "gold_yes": 0.5041}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 490

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Russian · atbench500 · en_yes

Solar Decide Flash, same named hosted route; Russian; slice atbench500. {"answered": 2005, "cases": 2005, "complete": true}

Source observations: {"n": 490, "acc": 0.498, "en_acc": 0.5041, "auroc": 0.5349, "en_auroc": 0.6287, "yes": 0.9163, "en_yes": 0.8612, "gold_yes": 0.5041}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 490

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Russian · jailbreak_classification · acc

Solar Decide Flash, same named hosted route; Russian; slice jailbreak_classification. {"answered": 2005, "cases": 2005, "complete": true}

Source observations: {"n": 399, "acc": 0.9323, "en_acc": 0.9699, "auroc": 0.9848, "en_auroc": 0.9883, "yes": 0.3985, "en_yes": 0.3509, "gold_yes": 0.3509}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 399

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Russian · jailbreak_classification · auroc

Solar Decide Flash, same named hosted route; Russian; slice jailbreak_classification. {"answered": 2005, "cases": 2005, "complete": true}

Source observations: {"n": 399, "acc": 0.9323, "en_acc": 0.9699, "auroc": 0.9848, "en_auroc": 0.9883, "yes": 0.3985, "en_yes": 0.3509, "gold_yes": 0.3509}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 399

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Russian · jailbreak_classification · en_acc

Solar Decide Flash, same named hosted route; Russian; slice jailbreak_classification. {"answered": 2005, "cases": 2005, "complete": true}

Source observations: {"n": 399, "acc": 0.9323, "en_acc": 0.9699, "auroc": 0.9848, "en_auroc": 0.9883, "yes": 0.3985, "en_yes": 0.3509, "gold_yes": 0.3509}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 399

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Russian · jailbreak_classification · en_auroc

Solar Decide Flash, same named hosted route; Russian; slice jailbreak_classification. {"answered": 2005, "cases": 2005, "complete": true}

Source observations: {"n": 399, "acc": 0.9323, "en_acc": 0.9699, "auroc": 0.9848, "en_auroc": 0.9883, "yes": 0.3985, "en_yes": 0.3509, "gold_yes": 0.3509}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 399

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Russian · jailbreak_classification · yes

Solar Decide Flash, same named hosted route; Russian; slice jailbreak_classification. {"answered": 2005, "cases": 2005, "complete": true}

Source observations: {"n": 399, "acc": 0.9323, "en_acc": 0.9699, "auroc": 0.9848, "en_auroc": 0.9883, "yes": 0.3985, "en_yes": 0.3509, "gold_yes": 0.3509}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 399

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Russian · jailbreak_classification · en_yes

Solar Decide Flash, same named hosted route; Russian; slice jailbreak_classification. {"answered": 2005, "cases": 2005, "complete": true}

Source observations: {"n": 399, "acc": 0.9323, "en_acc": 0.9699, "auroc": 0.9848, "en_auroc": 0.9883, "yes": 0.3985, "en_yes": 0.3509, "gold_yes": 0.3509}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 399

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Russian · prompt_injections · acc

Solar Decide Flash, same named hosted route; Russian; slice prompt_injections. {"answered": 2005, "cases": 2005, "complete": true}

Source observations: {"n": 116, "acc": 0.8276, "en_acc": 0.7931, "auroc": 0.9054, "en_auroc": 0.9051, "yes": 0.431, "en_yes": 0.3966, "gold_yes": 0.5172}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 116

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Russian · prompt_injections · auroc

Solar Decide Flash, same named hosted route; Russian; slice prompt_injections. {"answered": 2005, "cases": 2005, "complete": true}

Source observations: {"n": 116, "acc": 0.8276, "en_acc": 0.7931, "auroc": 0.9054, "en_auroc": 0.9051, "yes": 0.431, "en_yes": 0.3966, "gold_yes": 0.5172}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 116

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Russian · prompt_injections · en_acc

Solar Decide Flash, same named hosted route; Russian; slice prompt_injections. {"answered": 2005, "cases": 2005, "complete": true}

Source observations: {"n": 116, "acc": 0.8276, "en_acc": 0.7931, "auroc": 0.9054, "en_auroc": 0.9051, "yes": 0.431, "en_yes": 0.3966, "gold_yes": 0.5172}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 116

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Russian · prompt_injections · en_auroc

Solar Decide Flash, same named hosted route; Russian; slice prompt_injections. {"answered": 2005, "cases": 2005, "complete": true}

Source observations: {"n": 116, "acc": 0.8276, "en_acc": 0.7931, "auroc": 0.9054, "en_auroc": 0.9051, "yes": 0.431, "en_yes": 0.3966, "gold_yes": 0.5172}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 116

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Russian · prompt_injections · yes

Solar Decide Flash, same named hosted route; Russian; slice prompt_injections. {"answered": 2005, "cases": 2005, "complete": true}

Source observations: {"n": 116, "acc": 0.8276, "en_acc": 0.7931, "auroc": 0.9054, "en_auroc": 0.9051, "yes": 0.431, "en_yes": 0.3966, "gold_yes": 0.5172}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 116

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained

JevBench multilingual · Russian · prompt_injections · en_yes

Solar Decide Flash, same named hosted route; Russian; slice prompt_injections. {"answered": 2005, "cases": 2005, "complete": true}

Source observations: {"n": 116, "acc": 0.8276, "en_acc": 0.7931, "auroc": 0.9054, "en_auroc": 0.9051, "yes": 0.431, "en_yes": 0.3966, "gold_yes": 0.5172}

Translated binary benchmark, five slices; target-language and English answers use the same retained cases. Translation failures change per-language populations. Mean across five benchmark accuracies, not pooled item accuracy. AUROC is a native0–1 coefficient; delta is paired accuracy percentage points. Yes rate is response propensity, not quality. No Examenos inference or translation.

Checked / reported: 2026-11-02 · Samples: 116

Snapshot: 2026-10-16 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · Published summary observations; upstream task/result terms retained