Examenos

Four equally weighted evaluations: AA-Omniscience-Decision, HLE-MultiChoice (513 text-only questions), When2Call and WildGuardTest.

Exact named AA decision system; index v1.0. Served precision and detailed execution date not published in inspected chart.

Rounded chart label manually verified in source; not the ordinary language-model Intelligence Index.

Independent AA results. Omniscience splits its 25% between accuracy and non-hallucination. Displayed integer chart values, not undisclosed precise values. No automated AA import or refresh adapter.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-17 · Revision: Manually checked chart · 2026-10-17 · Published observations; result reuse terms not specified

400 contrastive decisions in 200 pairs across eight workflow tasks; pair accuracy needs both answers correct.

{"model_requested": "gpt-6-luna", "model_served": "gpt-6-luna", "endpoint": "OpenAI /v1/decisions (public beta)", "protocol": "few-shot: one example per option", "client_region": null, "concurrency": 4, "warmup_calls": 3, "harness_commit": "decidebench@c4913b2"}

Run 2026-10-08. Workload cost is per 1M tasks, not a token tariff.

Synthetic Claude-written items, structural validation and label spot checks. Few-shot examples except explicit zero-shot column. Hosted costs use observed billed usage and dated tariffs. Hosted latency includes provider/gateway, four concurrent calls and three warmups. Luna timing client differs from the other hosted runs, so latency is not a controlled speed ranking.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-08 · Samples: 400

Snapshot: 2026-10-17 · Revision: bbf8f2c77b544952b42be555866729a22c33980d · CC BY 4.0 original dataset; MIT code/report

400 contrastive decisions in 200 pairs across eight workflow tasks; pair accuracy needs both answers correct.

{"model_requested": "gpt-6-luna", "model_served": "gpt-6-luna", "endpoint": "OpenAI /v1/decisions (public beta)", "protocol": "few-shot: one example per option", "client_region": null, "concurrency": 4, "warmup_calls": 3, "harness_commit": "decidebench@c4913b2"}

Run 2026-10-08. Workload cost is per 1M tasks, not a token tariff.

Synthetic Claude-written items, structural validation and label spot checks. Few-shot examples except explicit zero-shot column. Hosted costs use observed billed usage and dated tariffs. Hosted latency includes provider/gateway, four concurrent calls and three warmups. Luna timing client differs from the other hosted runs, so latency is not a controlled speed ranking.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-08 · Samples: 50

Snapshot: 2026-10-17 · Revision: bbf8f2c77b544952b42be555866729a22c33980d · CC BY 4.0 original dataset; MIT code/report

400 contrastive decisions in 200 pairs across eight workflow tasks; pair accuracy needs both answers correct.

{"model_requested": "gpt-6-luna", "model_served": "gpt-6-luna", "endpoint": "OpenAI /v1/decisions (public beta)", "protocol": "few-shot: one example per option", "client_region": null, "concurrency": 4, "warmup_calls": 3, "harness_commit": "decidebench@c4913b2"}

Run 2026-10-08. Workload cost is per 1M tasks, not a token tariff.

Synthetic Claude-written items, structural validation and label spot checks. Few-shot examples except explicit zero-shot column. Hosted costs use observed billed usage and dated tariffs. Hosted latency includes provider/gateway, four concurrent calls and three warmups. Luna timing client differs from the other hosted runs, so latency is not a controlled speed ranking.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-08 · Samples: 50

Snapshot: 2026-10-17 · Revision: bbf8f2c77b544952b42be555866729a22c33980d · CC BY 4.0 original dataset; MIT code/report

400 contrastive decisions in 200 pairs across eight workflow tasks; pair accuracy needs both answers correct.

{"model_requested": "gpt-6-luna", "model_served": "gpt-6-luna", "endpoint": "OpenAI /v1/decisions (public beta)", "protocol": "few-shot: one example per option", "client_region": null, "concurrency": 4, "warmup_calls": 3, "harness_commit": "decidebench@c4913b2"}

Run 2026-10-08. Workload cost is per 1M tasks, not a token tariff.

Synthetic Claude-written items, structural validation and label spot checks. Few-shot examples except explicit zero-shot column. Hosted costs use observed billed usage and dated tariffs. Hosted latency includes provider/gateway, four concurrent calls and three warmups. Luna timing client differs from the other hosted runs, so latency is not a controlled speed ranking.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-08 · Samples: 50

Snapshot: 2026-10-17 · Revision: bbf8f2c77b544952b42be555866729a22c33980d · CC BY 4.0 original dataset; MIT code/report

400 contrastive decisions in 200 pairs across eight workflow tasks; pair accuracy needs both answers correct.

{"model_requested": "gpt-6-luna", "model_served": "gpt-6-luna", "endpoint": "OpenAI /v1/decisions (public beta)", "protocol": "few-shot: one example per option", "client_region": null, "concurrency": 4, "warmup_calls": 3, "harness_commit": "decidebench@c4913b2"}

Run 2026-10-08. Workload cost is per 1M tasks, not a token tariff.

Synthetic Claude-written items, structural validation and label spot checks. Few-shot examples except explicit zero-shot column. Hosted costs use observed billed usage and dated tariffs. Hosted latency includes provider/gateway, four concurrent calls and three warmups. Luna timing client differs from the other hosted runs, so latency is not a controlled speed ranking.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-08 · Samples: 50

Snapshot: 2026-10-17 · Revision: bbf8f2c77b544952b42be555866729a22c33980d · CC BY 4.0 original dataset; MIT code/report

400 contrastive decisions in 200 pairs across eight workflow tasks; pair accuracy needs both answers correct.

{"model_requested": "gpt-6-luna", "model_served": "gpt-6-luna", "endpoint": "OpenAI /v1/decisions (public beta)", "protocol": "few-shot: one example per option", "client_region": null, "concurrency": 4, "warmup_calls": 3, "harness_commit": "decidebench@c4913b2"}

Run 2026-10-08. Workload cost is per 1M tasks, not a token tariff.

Synthetic Claude-written items, structural validation and label spot checks. Few-shot examples except explicit zero-shot column. Hosted costs use observed billed usage and dated tariffs. Hosted latency includes provider/gateway, four concurrent calls and three warmups. Luna timing client differs from the other hosted runs, so latency is not a controlled speed ranking.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-08 · Samples: 400

Snapshot: 2026-10-17 · Revision: bbf8f2c77b544952b42be555866729a22c33980d · CC BY 4.0 original dataset; MIT code/report

400 contrastive decisions in 200 pairs across eight workflow tasks; pair accuracy needs both answers correct.

{"model_requested": "gpt-6-luna", "model_served": "gpt-6-luna", "endpoint": "OpenAI /v1/decisions (public beta)", "protocol": "few-shot: one example per option", "client_region": null, "concurrency": 4, "warmup_calls": 3, "harness_commit": "decidebench@c4913b2"}

Run 2026-10-08. Workload cost is per 1M tasks, not a token tariff.

Synthetic Claude-written items, structural validation and label spot checks. Few-shot examples except explicit zero-shot column. Hosted costs use observed billed usage and dated tariffs. Hosted latency includes provider/gateway, four concurrent calls and three warmups. Luna timing client differs from the other hosted runs, so latency is not a controlled speed ranking.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-08 · Samples: 400

Snapshot: 2026-10-17 · Revision: bbf8f2c77b544952b42be555866729a22c33980d · CC BY 4.0 original dataset; MIT code/report

400 contrastive decisions in 200 pairs across eight workflow tasks; pair accuracy needs both answers correct.

{"model_requested": "gpt-6-luna", "model_served": "gpt-6-luna", "endpoint": "OpenAI /v1/decisions (public beta)", "protocol": "few-shot: one example per option", "client_region": null, "concurrency": 4, "warmup_calls": 3, "harness_commit": "decidebench@c4913b2"}

Run 2026-10-08. Workload cost is per 1M tasks, not a token tariff.

Synthetic Claude-written items, structural validation and label spot checks. Few-shot examples except explicit zero-shot column. Hosted costs use observed billed usage and dated tariffs. Hosted latency includes provider/gateway, four concurrent calls and three warmups. Luna timing client differs from the other hosted runs, so latency is not a controlled speed ranking.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-08 · Samples: 200

Snapshot: 2026-10-17 · Revision: bbf8f2c77b544952b42be555866729a22c33980d · CC BY 4.0 original dataset; MIT code/report

400 contrastive decisions in 200 pairs across eight workflow tasks; pair accuracy needs both answers correct.

{"model_requested": "gpt-6-luna", "model_served": "gpt-6-luna", "endpoint": "OpenAI /v1/decisions (public beta)", "protocol": "few-shot: one example per option", "client_region": null, "concurrency": 4, "warmup_calls": 3, "harness_commit": "decidebench@c4913b2"}

Run 2026-10-08. Workload cost is per 1M tasks, not a token tariff.

Synthetic Claude-written items, structural validation and label spot checks. Few-shot examples except explicit zero-shot column. Hosted costs use observed billed usage and dated tariffs. Hosted latency includes provider/gateway, four concurrent calls and three warmups. Luna timing client differs from the other hosted runs, so latency is not a controlled speed ranking.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-08 · Samples: 50

Snapshot: 2026-10-17 · Revision: bbf8f2c77b544952b42be555866729a22c33980d · CC BY 4.0 original dataset; MIT code/report

400 contrastive decisions in 200 pairs across eight workflow tasks; pair accuracy needs both answers correct.

{"model_requested": "gpt-6-luna", "model_served": "gpt-6-luna", "endpoint": "OpenAI /v1/decisions (public beta)", "protocol": "few-shot: one example per option", "client_region": null, "concurrency": 4, "warmup_calls": 3, "harness_commit": "decidebench@c4913b2"}

Run 2026-10-08. Workload cost is per 1M tasks, not a token tariff.

Synthetic Claude-written items, structural validation and label spot checks. Few-shot examples except explicit zero-shot column. Hosted costs use observed billed usage and dated tariffs. Hosted latency includes provider/gateway, four concurrent calls and three warmups. Luna timing client differs from the other hosted runs, so latency is not a controlled speed ranking.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-08 · Samples: 50

Snapshot: 2026-10-17 · Revision: bbf8f2c77b544952b42be555866729a22c33980d · CC BY 4.0 original dataset; MIT code/report

400 contrastive decisions in 200 pairs across eight workflow tasks; pair accuracy needs both answers correct.

{"model_requested": "gpt-6-luna", "model_served": "gpt-6-luna", "endpoint": "OpenAI /v1/decisions (public beta)", "protocol": "few-shot: one example per option", "client_region": null, "concurrency": 4, "warmup_calls": 3, "harness_commit": "decidebench@c4913b2"}

Run 2026-10-08. Workload cost is per 1M tasks, not a token tariff.

Synthetic Claude-written items, structural validation and label spot checks. Few-shot examples except explicit zero-shot column. Hosted costs use observed billed usage and dated tariffs. Hosted latency includes provider/gateway, four concurrent calls and three warmups. Luna timing client differs from the other hosted runs, so latency is not a controlled speed ranking.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-08 · Samples: 50

Snapshot: 2026-10-17 · Revision: bbf8f2c77b544952b42be555866729a22c33980d · CC BY 4.0 original dataset; MIT code/report

400 contrastive decisions in 200 pairs across eight workflow tasks; pair accuracy needs both answers correct.

{"model_requested": "gpt-6-luna", "model_served": "gpt-6-luna", "endpoint": "OpenAI /v1/decisions (public beta)", "protocol": "few-shot: one example per option", "client_region": null, "concurrency": 4, "warmup_calls": 3, "harness_commit": "decidebench@c4913b2"}

Run 2026-10-08. Workload cost is per 1M tasks, not a token tariff.

Synthetic Claude-written items, structural validation and label spot checks. Few-shot examples except explicit zero-shot column. Hosted costs use observed billed usage and dated tariffs. Hosted latency includes provider/gateway, four concurrent calls and three warmups. Luna timing client differs from the other hosted runs, so latency is not a controlled speed ranking.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-08 · Samples: 50

Snapshot: 2026-10-17 · Revision: bbf8f2c77b544952b42be555866729a22c33980d · CC BY 4.0 original dataset; MIT code/report

400 contrastive decisions in 200 pairs across eight workflow tasks; pair accuracy needs both answers correct.

{"model_requested": "gpt-6-luna", "model_served": "gpt-6-luna", "endpoint": "OpenAI /v1/decisions (public beta)", "protocol": "few-shot: one example per option", "client_region": null, "concurrency": 4, "warmup_calls": 3, "harness_commit": "decidebench@c4913b2"}

Run 2026-10-08. Workload cost is per 1M tasks, not a token tariff.

Synthetic Claude-written items, structural validation and label spot checks. Few-shot examples except explicit zero-shot column. Hosted costs use observed billed usage and dated tariffs. Hosted latency includes provider/gateway, four concurrent calls and three warmups. Luna timing client differs from the other hosted runs, so latency is not a controlled speed ranking.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-08 · Samples: 400

Snapshot: 2026-10-17 · Revision: bbf8f2c77b544952b42be555866729a22c33980d · CC BY 4.0 original dataset; MIT code/report

JevBench Binary slices: aegis2 · acc @0.5

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 74.8%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: aegis2 · acc if flipped

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 25.2%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: aegis2 · AUROC

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.8426.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: aegis2 · Brier

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.2342.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: aegis2 · ECE

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.2288.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: aegis2 · FPR

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 1.6%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: aegis2 · positive recall

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 51.2%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: aegis2_response · acc @0.5

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 72.6%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: aegis2_response · acc if flipped

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 27.4%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: aegis2_response · AUROC

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.8496.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: aegis2_response · Brier

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.2431.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: aegis2_response · ECE

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.2238.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: aegis2_response · FPR

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 5.2%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: aegis2_response · positive recall

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 50.4%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: atbench500 · acc @0.5

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 87.8%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: atbench500 · acc if flipped

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 12.2%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: atbench500 · AUROC

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.9710.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: atbench500 · Brier

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.0951.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: atbench500 · ECE

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.0681.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: atbench500 · FPR

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 24.4%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: atbench500 · positive recall

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 100.0%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: jailbreak_classification · acc @0.5

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 96.8%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 400

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: jailbreak_classification · acc if flipped

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 3.2%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 400

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: jailbreak_classification · AUROC

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.9782.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 400

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: jailbreak_classification · Brier

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.0277.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 400

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: jailbreak_classification · ECE

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.0228.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 400

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: jailbreak_classification · FPR

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.4%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 400

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: jailbreak_classification · positive recall

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 91.5%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 400

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: prompt_injections · acc @0.5

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 63.8%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 116

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: prompt_injections · acc if flipped

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 36.2%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 116

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: prompt_injections · AUROC

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.6708.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 116

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: prompt_injections · Brier

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.3275.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 116

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: prompt_injections · ECE

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.3086.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 116

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: prompt_injections · FPR

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.0%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 116

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Binary slices: prompt_injections · positive recall

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 30.0%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 116

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench By subset: rewardbench2 · accuracy

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 82.8%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 238

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench By subset: rewardbench2 · accuracy

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 91.5%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 236

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench By subset: rewardbench2 · accuracy

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 63.1%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 236

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench By subset: rewardbench2 · accuracy

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 37.7%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 239

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench By subset: rewardbench2 · accuracy

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 91.9%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 222

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: banking77 · accuracy

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 78.6%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 1000

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: banking77 · Brier

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.3301.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 1000

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: banking77 · ECE (10 bins)

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.0798.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 1000

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: banking77 · macro-F1

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.7704.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 1000

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: banking77 · NLL

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.5726.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 1000

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: mcpbench_server_routing · accuracy

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 96.4%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 110

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: mcpbench_server_routing · Brier

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.0634.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 110

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: mcpbench_server_routing · ECE (10 bins)

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.0689.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 110

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: mcpbench_server_routing · macro-F1

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.9631.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 110

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: mcpbench_server_routing · NLL

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.1523.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 110

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: mcpbench_tool_selection · accuracy

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 100.0%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 26

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: mcpbench_tool_selection · Brier

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.0017.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 26

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: mcpbench_tool_selection · ECE (10 bins)

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.0185.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 26

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: mcpbench_tool_selection · macro-F1

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 1.0000.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 26

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: mcpbench_tool_selection · NLL

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.0191.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 26

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: mcpmark_server_routing · accuracy

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 49.1%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 336

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: mcpmark_server_routing · Brier

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.6792.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 336

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: mcpmark_server_routing · ECE (10 bins)

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.1565.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 336

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: mcpmark_server_routing · macro-F1

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.4268.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 336

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: mcpmark_server_routing · NLL

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 1.4334.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 336

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: mcpmark_tool_selection · accuracy

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 100.0%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 18

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: mcpmark_tool_selection · Brier

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.0000.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 18

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: mcpmark_tool_selection · ECE (10 bins)

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.0022.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 18

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: mcpmark_tool_selection · macro-F1

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 1.0000.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 18

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: mcpmark_tool_selection · NLL

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.0022.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 18

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: medmcqa · accuracy

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 77.1%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 986

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: medmcqa · Brier

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.3262.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 986

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: medmcqa · ECE (10 bins)

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.0812.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 986

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: medmcqa · macro-F1

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.7707.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 986

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: medmcqa · NLL

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.5967.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 986

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: medqa_usmle · accuracy

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 87.6%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 999

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: medqa_usmle · Brier

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.1779.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 999

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: medqa_usmle · ECE (10 bins)

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.0508.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 999

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: medqa_usmle · macro-F1

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.8762.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 999

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: medqa_usmle · NLL

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.3143.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 999

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: mmlu_pro · accuracy

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 59.9%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 995

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: mmlu_pro · Brier

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.5356.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 995

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: mmlu_pro · ECE (10 bins)

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.0701.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 995

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: mmlu_pro · macro-F1

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.5958.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 995

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: mmlu_pro · NLL

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 1.2030.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 995

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: pubmedqa · accuracy

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 62.8%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: pubmedqa · Brier

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.6238.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: pubmedqa · ECE (10 bins)

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.2628.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: pubmedqa · macro-F1

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.5861.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: pubmedqa · NLL

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.8298.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: rewardbench2 · accuracy

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 73.1%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 1171

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: rewardbench2 · Brier

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.3736.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 1171

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: rewardbench2 · ECE (10 bins)

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.0304.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 1171

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: rewardbench2 · macro-F1

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.7298.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 1171

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: rewardbench2 · NLL

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.7246.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 1171

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: scienceqa_text · accuracy

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 94.0%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 1000

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: scienceqa_text · Brier

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.0760.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 1000

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: scienceqa_text · ECE (10 bins)

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.0144.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 1000

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: scienceqa_text · macro-F1

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.9580.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 1000

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Choice slices: scienceqa_text · NLL

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.1208.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 1000

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Headline: aegis2 · accuracy

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.7480.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 500 · Source 95% CI: 71–78.60000000000001

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Headline: aegis2_response · accuracy

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.7260.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 500 · Source 95% CI: 68.60000000000001–76.6

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Headline: atbench500 · accuracy

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.8780.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 500 · Source 95% CI: 85–90.60000000000001

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Headline: banking77 · accuracy

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.7860.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 1000 · Source 95% CI: 75.3–81.8

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Headline: helpsteer2_helpfulness · mae_expected_level

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.9673.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 500 · Source 95% CI: 0.9–1.036

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Headline: jailbreak_classification · accuracy

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.9675.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 400 · Source 95% CI: 95–98.5

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Headline: mcpbench_server_routing · server_routing_accuracy

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.9636.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 112 · Source 95% CI: 90.9–100

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Headline: mcpbench_tool_selection · tool_selection_accuracy

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 1.0000.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 26

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Headline: mcpmark_server_routing · server_routing_accuracy

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.4911.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 338 · Source 95% CI: 42.3–56.00000000000001

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Headline: mcpmark_tool_selection · tool_selection_accuracy

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 1.0000.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 18

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Headline: medmcqa · accuracy

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.7708.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 1000 · Source 95% CI: 73.7–80.4

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Headline: medqa_usmle · accuracy

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.8759.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 1000 · Source 95% CI: 84.89999999999999–90.3

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Headline: mmlu_pro · accuracy

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.5990.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 1000 · Source 95% CI: 56.00000000000001–63.7

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Headline: nfcorpus_rank_choice · ndcg@10

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.3531.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 298 · Source 95% CI: 31.7–38.9

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Headline: nfcorpus_rank_graded · ndcg@10

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.3648.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 4924 · Source 95% CI: 33.2–40.1

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Headline: nfcorpus_rank_noul · ndcg@10

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.3676.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 298 · Source 95% CI: 33.1–40.300000000000004

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Headline: prompt_injections · accuracy

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.6379.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 116 · Source 95% CI: 55.2–72.39999999999999

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Headline: pubmedqa · accuracy

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.6280.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 500 · Source 95% CI: 58.599999999999994–66.8

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Headline: rewardbench2 · accuracy

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.7310.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 1200 · Source 95% CI: 69.89999999999999–76.3

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Headline: scienceqa_text · accuracy

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.9400.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 1000 · Source 95% CI: 92.2–95.7

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Headline: scifact_rank_choice · ndcg@10

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.7444.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 300 · Source 95% CI: 70.19999999999999–78.7

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Headline: scifact_rank_noul · ndcg@10

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.7293.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 300 · Source 95% CI: 68.2–77.60000000000001

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Headline: sst5 · mae_expected_level

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.5424.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 500 · Source 95% CI: 0.498–0.587

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: banking77 · both correct

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 74.2%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: banking77 · flipped by reordering

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 8.8%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: banking77 · original order

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 79.6%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: banking77 · permuted order

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 77.6%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: mcpbench_server_routing · both correct

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 96.4%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 55

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: mcpbench_server_routing · flipped by reordering

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.0%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 55

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: mcpbench_server_routing · original order

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 96.4%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 55

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: mcpbench_server_routing · permuted order

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 96.4%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 55

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: mcpbench_tool_selection · both correct

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 100.0%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 4

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: mcpbench_tool_selection · flipped by reordering

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.0%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 4

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: mcpbench_tool_selection · original order

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 100.0%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 4

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: mcpbench_tool_selection · permuted order

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 100.0%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 4

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: mcpmark_server_routing · both correct

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 39.9%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 168

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: mcpmark_server_routing · flipped by reordering

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 18.5%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 168

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: mcpmark_server_routing · original order

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 47.6%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 168

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: mcpmark_server_routing · permuted order

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 50.6%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 168

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: mcpmark_tool_selection · both correct

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 100.0%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 3

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: mcpmark_tool_selection · flipped by reordering

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.0%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 3

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: mcpmark_tool_selection · original order

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 100.0%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 3

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: mcpmark_tool_selection · permuted order

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 100.0%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 3

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: medmcqa · both correct

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 71.7%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 491

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: medmcqa · flipped by reordering

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 11.0%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 491

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: medmcqa · original order

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 78.3%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 491

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: medmcqa · permuted order

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 75.9%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 491

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: medqa_usmle · both correct

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 84.8%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 499

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: medqa_usmle · flipped by reordering

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 5.8%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 499

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: medqa_usmle · original order

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 88.4%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 499

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: medqa_usmle · permuted order

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 86.8%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 499

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: mmlu_pro · both correct

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 52.7%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 497

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: mmlu_pro · flipped by reordering

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 14.5%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 497

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: mmlu_pro · original order

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 60.8%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 497

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: mmlu_pro · permuted order

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 59.0%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 497

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: rewardbench2 · both correct

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 63.5%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 573

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: rewardbench2 · flipped by reordering

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 18.8%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 573

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: rewardbench2 · original order

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 73.2%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 573

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: rewardbench2 · permuted order

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 73.0%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 573

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: scienceqa_text · both correct

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 90.8%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: scienceqa_text · flipped by reordering

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 6.4%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: scienceqa_text · original order

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 94.6%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Option-order sensitivity: scienceqa_text · permuted order

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 93.4%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ordinal slices: helpsteer2_helpfulness · exact (rounded)

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 32.3%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 499

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ordinal slices: helpsteer2_helpfulness · MAE (expected level)

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.967.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 499

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ordinal slices: helpsteer2_helpfulness · RPS

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.1928.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 499

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ordinal slices: helpsteer2_helpfulness · Spearman

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.492.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 499

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ordinal slices: helpsteer2_helpfulness · within 1 level

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 78.0%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 499

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ordinal slices: nfcorpus_rank_graded · exact (rounded)

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 54.5%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 4924

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ordinal slices: nfcorpus_rank_graded · MAE (expected level)

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.507.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 4924

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ordinal slices: nfcorpus_rank_graded · RPS

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.1818.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 4924

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ordinal slices: nfcorpus_rank_graded · Spearman

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.488.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 4924

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ordinal slices: nfcorpus_rank_graded · within 1 level

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 95.3%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 4924

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ordinal slices: sst5 · exact (rounded)

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 53.6%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ordinal slices: sst5 · MAE (expected level)

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.542.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ordinal slices: sst5 · RPS

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.1028.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ordinal slices: sst5 · Spearman

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.849.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ordinal slices: sst5 · within 1 level

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 96.0%.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 500

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ranking slices: nfcorpus_rank_choice · MRR@10

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.6146.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 283

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ranking slices: nfcorpus_rank_choice · nDCG@10

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.3531.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 283

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ranking slices: nfcorpus_rank_choice · Recall@10

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.1624.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 283

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ranking slices: nfcorpus_rank_graded · MRR@10

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.6087.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 298

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ranking slices: nfcorpus_rank_graded · nDCG@10

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.3648.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 298

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ranking slices: nfcorpus_rank_graded · Recall@10

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.1698.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 298

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ranking slices: nfcorpus_rank_noul · MRR@10

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.6113.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 298

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ranking slices: nfcorpus_rank_noul · nDCG@10

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.3676.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 298

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ranking slices: nfcorpus_rank_noul · Recall@10

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.1708.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 298

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ranking slices: scifact_rank_choice · MRR@10

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.7426.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 299

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ranking slices: scifact_rank_choice · nDCG@10

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.7444.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 299

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ranking slices: scifact_rank_choice · Recall@10

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.8008.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 299

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ranking slices: scifact_rank_noul · MRR@10

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.7191.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 300

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ranking slices: scifact_rank_noul · nDCG@10

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.7293.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 300

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

JevBench Ranking slices: scifact_rank_noul · Recall@10

Scored 2026-10-15T23:39:27-0700. Model requested `or-openai-gpt-6-luna-decisions`, endpoint `https://openrouter.ai/api/alpha/decisions`, manifest pinned 2026-09-26. One typed question per request, except `*_rank_noul` slices, which ask one noul per candidate document in a single request.

Raw 0.7976.

Named source slice, frozen manifest 26 September. One typed question per request except per-document ranking. Unrun is missing; API rejection handling follows source. Bootstrap intervals group reordered twins; ECE uses 10 equal-width bins. Choice Brier is sum convention, binary Brier is separate. NLL excludes exactly-zero gold mass. Ordinal level distances are assumed equal. Option-order sensitivity is correctness flips, not probability drift. Rankings use fixed BM25 top-20. Different direct/gateway d1 reports remain separate.

Checked / reported: 2026-10-17 · Evaluated: 2026-10-15T23:39:27-0700 · Samples: 300

Snapshot: 2026-10-17 · Revision: 6954ce9d98d81210d043ca0e94bc72bff6443140 · MIT code and released reports; source cases retain upstream licenses

Liquid vendor’s 112 unique evaluations grouped as Average; not an independent index.

Liquid vendor-run October 15 panel. Exact d1-ultra/Jev 1.14/d3 labels; remaining endpoint revisions and served precision unspecified.

Figure-only printed quantity. Source does not publish per-task weights, complete item populations or all inference settings. Average is the vendor’s published aggregate, never recomputed from rounded domains. Peer model exact hosted revisions can be unspecified.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-15 · Revision: Release figures 2026-10-15 · Published observations; result reuse terms not specified

Liquid vendor’s 112 unique evaluations grouped as Business and new fields; not an independent index.

Liquid vendor-run October 15 panel. Exact d1-ultra/Jev 1.14/d3 labels; remaining endpoint revisions and served precision unspecified.

Figure-only printed quantity. Source does not publish per-task weights, complete item populations or all inference settings. Average is the vendor’s published aggregate, never recomputed from rounded domains. Peer model exact hosted revisions can be unspecified.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-15 · Revision: Release figures 2026-10-15 · Published observations; result reuse terms not specified

Liquid vendor’s 112 unique evaluations grouped as Judgment and taste; not an independent index.

Liquid vendor-run October 15 panel. Exact d1-ultra/Jev 1.14/d3 labels; remaining endpoint revisions and served precision unspecified.

Figure-only printed quantity. Source does not publish per-task weights, complete item populations or all inference settings. Average is the vendor’s published aggregate, never recomputed from rounded domains. Peer model exact hosted revisions can be unspecified.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-15 · Revision: Release figures 2026-10-15 · Published observations; result reuse terms not specified

Liquid vendor’s 112 unique evaluations grouped as Knowledge and reasoning; not an independent index.

Liquid vendor-run October 15 panel. Exact d1-ultra/Jev 1.14/d3 labels; remaining endpoint revisions and served precision unspecified.

Figure-only printed quantity. Source does not publish per-task weights, complete item populations or all inference settings. Average is the vendor’s published aggregate, never recomputed from rounded domains. Peer model exact hosted revisions can be unspecified.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-15 · Revision: Release figures 2026-10-15 · Published observations; result reuse terms not specified

Liquid vendor’s 112 unique evaluations grouped as Languages; not an independent index.

Liquid vendor-run October 15 panel. Exact d1-ultra/Jev 1.14/d3 labels; remaining endpoint revisions and served precision unspecified.

Figure-only printed quantity. Source does not publish per-task weights, complete item populations or all inference settings. Average is the vendor’s published aggregate, never recomputed from rounded domains. Peer model exact hosted revisions can be unspecified.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-15 · Revision: Release figures 2026-10-15 · Published observations; result reuse terms not specified

Liquid vendor’s 112 unique evaluations grouped as Robustness; not an independent index.

Liquid vendor-run October 15 panel. Exact d1-ultra/Jev 1.14/d3 labels; remaining endpoint revisions and served precision unspecified.

Figure-only printed quantity. Source does not publish per-task weights, complete item populations or all inference settings. Average is the vendor’s published aggregate, never recomputed from rounded domains. Peer model exact hosted revisions can be unspecified.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-15 · Revision: Release figures 2026-10-15 · Published observations; result reuse terms not specified

Liquid vendor’s 112 unique evaluations grouped as Safety and policy; not an independent index.

Liquid vendor-run October 15 panel. Exact d1-ultra/Jev 1.14/d3 labels; remaining endpoint revisions and served precision unspecified.

Figure-only printed quantity. Source does not publish per-task weights, complete item populations or all inference settings. Average is the vendor’s published aggregate, never recomputed from rounded domains. Peer model exact hosted revisions can be unspecified.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-15 · Revision: Release figures 2026-10-15 · Published observations; result reuse terms not specified

Liquid vendor’s 112 unique evaluations grouped as Tools and agents; not an independent index.

Liquid vendor-run October 15 panel. Exact d1-ultra/Jev 1.14/d3 labels; remaining endpoint revisions and served precision unspecified.

Figure-only printed quantity. Source does not publish per-task weights, complete item populations or all inference settings. Average is the vendor’s published aggregate, never recomputed from rounded domains. Peer model exact hosted revisions can be unspecified.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-15 · Revision: Release figures 2026-10-15 · Published observations; result reuse terms not specified

Vendor-defined mean of absolute accuracy minus chosen confidence; not interchangeable with other studies’ ECE.

Liquid vendor figure: mean absolute accuracy−confidence on chosen option; hardware/precision unpublished.

Printed figure values manually inspected. Curve coordinates are not reconstructed. Source task weighting and binning are not fully published.

Checked / reported: 2026-10-17 · Samples: Not published

Snapshot: 2026-10-15 · Revision: Release figure 2026-10-15 · Published observations; result reuse terms not specified

Exact named hosted API observation; All-case accuracy.

Named source route openai/gpt-6-luna-decisions; ModelTested runner v1; tested_on=openrouter. Exact backend weight revision and API inference precision unpublished.

Publisher generated 2026-10-18. Execution dates, token usage and hashes remain in its per-run receipts. No private prompt/response archive imported.

56 authored situations, including 19 hard cases, each with two typed questions; 14 public and 42 hidden cases. Source aggregate averages per-case decision accuracy. Confidence score is 100×(1−mean multiclass Brier/2), not ECE or a calibration probability. Small single-run benchmark; current HF access/size does not inherit this evaluator’s stale closed/API-only metadata. No tariff inferred from rounded cost 0.0.

Checked / reported: 2026-10-18 · Samples: 56

Snapshot: 2026-10-18 · Revision: 081e8f37130a4a9077b8cffe62827d4c3b9e3b23 · Published aggregate observations; result reuse terms not separately specified

Exact named hosted API observation; Hard-case accuracy.

Named source route openai/gpt-6-luna-decisions; ModelTested runner v1; tested_on=openrouter. Exact backend weight revision and API inference precision unpublished.

Publisher generated 2026-10-18. Execution dates, token usage and hashes remain in its per-run receipts. No private prompt/response archive imported.

56 authored situations, including 19 hard cases, each with two typed questions; 14 public and 42 hidden cases. Source aggregate averages per-case decision accuracy. Confidence score is 100×(1−mean multiclass Brier/2), not ECE or a calibration probability. Small single-run benchmark; current HF access/size does not inherit this evaluator’s stale closed/API-only metadata. No tariff inferred from rounded cost 0.0.

Checked / reported: 2026-10-18 · Samples: 19

Snapshot: 2026-10-18 · Revision: 081e8f37130a4a9077b8cffe62827d4c3b9e3b23 · Published aggregate observations; result reuse terms not separately specified

Exact named hosted API observation; Public-case accuracy.

Named source route openai/gpt-6-luna-decisions; ModelTested runner v1; tested_on=openrouter. Exact backend weight revision and API inference precision unpublished.

Publisher generated 2026-10-18. Execution dates, token usage and hashes remain in its per-run receipts. No private prompt/response archive imported.

56 authored situations, including 19 hard cases, each with two typed questions; 14 public and 42 hidden cases. Source aggregate averages per-case decision accuracy. Confidence score is 100×(1−mean multiclass Brier/2), not ECE or a calibration probability. Small single-run benchmark; current HF access/size does not inherit this evaluator’s stale closed/API-only metadata. No tariff inferred from rounded cost 0.0.

Checked / reported: 2026-10-18 · Samples: 14

Snapshot: 2026-10-18 · Revision: 081e8f37130a4a9077b8cffe62827d4c3b9e3b23 · Published aggregate observations; result reuse terms not separately specified

Exact named hosted API observation; Hidden-case accuracy.

Named source route openai/gpt-6-luna-decisions; ModelTested runner v1; tested_on=openrouter. Exact backend weight revision and API inference precision unpublished.

Publisher generated 2026-10-18. Execution dates, token usage and hashes remain in its per-run receipts. No private prompt/response archive imported.

56 authored situations, including 19 hard cases, each with two typed questions; 14 public and 42 hidden cases. Source aggregate averages per-case decision accuracy. Confidence score is 100×(1−mean multiclass Brier/2), not ECE or a calibration probability. Small single-run benchmark; current HF access/size does not inherit this evaluator’s stale closed/API-only metadata. No tariff inferred from rounded cost 0.0.

Checked / reported: 2026-10-18 · Samples: 42

Snapshot: 2026-10-18 · Revision: 081e8f37130a4a9077b8cffe62827d4c3b9e3b23 · Published aggregate observations; result reuse terms not separately specified

Exact named hosted API observation; Brier-based confidence score.

Named source route openai/gpt-6-luna-decisions; ModelTested runner v1; tested_on=openrouter. Exact backend weight revision and API inference precision unpublished.

Publisher generated 2026-10-18. Execution dates, token usage and hashes remain in its per-run receipts. No private prompt/response archive imported.

56 authored situations, including 19 hard cases, each with two typed questions; 14 public and 42 hidden cases. Source aggregate averages per-case decision accuracy. Confidence score is 100×(1−mean multiclass Brier/2), not ECE or a calibration probability. Small single-run benchmark; current HF access/size does not inherit this evaluator’s stale closed/API-only metadata. No tariff inferred from rounded cost 0.0.

Checked / reported: 2026-10-18 · Samples: 56

Snapshot: 2026-10-18 · Revision: 081e8f37130a4a9077b8cffe62827d4c3b9e3b23 · Published aggregate observations; result reuse terms not separately specified