Examenos

59.1% · raw 59.1 %

Headline · max effort · Own vendor

Source/record date: 2026-10-12

Harness: Meta launch evaluation; Internal seven-suite composite; official Meta PSI, not an independent index. Source checked 12 October 2026; evaluation date unpublished. Transcribed from model-card page 3 image; max effort except Full SWE-Marathon xhigh.

13.1% · raw 13.1 %

Headline · max effort · Own vendor

Source/record date: 2026-10-12

Harness: Meta launch evaluation; Vendor cites AA v1.1 hallucination-gated all-pass rate. Direct AA Harvey board checked separately; new Wonder configurations not yet in its 25-row table. Source checked 12 October 2026; evaluation date unpublished. Transcribed from model-card page 3 image; max effort except Full SWE-Marathon xhigh.

59.1% · raw 59.1 %

Headline · max effort · Own vendor

Source/record date: 2026-10-12

Harness: Meta launch evaluation; Vendor cites Vals private-test-set accuracy; independently checked Vals values retain full published precision in evaluators.json. Source checked 12 October 2026; evaluation date unpublished. Transcribed from model-card page 3 image; max effort except Full SWE-Marathon xhigh.

62.3% · raw 62.3 %

Headline · max effort · Own vendor

Source/record date: 2026-10-12

Harness: Meta launch evaluation; 525 professional health questions; no tools, GPT-5.4 rubric grader, length-normalized weighted rubric score. Claude scores cited from their Anthropic system cards. Source checked 12 October 2026; evaluation date unpublished. Transcribed from model-card page 3 image; max effort except Full SWE-Marathon xhigh.

45.4% · raw 45.4 %

Headline · max effort · Own vendor

Source/record date: 2026-10-12

Harness: Meta launch evaluation; Vendor-cited strict pass@1: every deterministic end-state assertion must pass. Not AA model-page partial reward (68.09/71.33 for Wonder/Full). Source checked 12 October 2026; evaluation date unpublished. Transcribed from model-card page 3 image; max effort except Full SWE-Marathon xhigh.

33.2% · raw 33.2 %

Headline · max effort · Own vendor

Source/record date: 2026-10-12

Harness: Meta launch evaluation; Vendor cites Surge professional PDF work score. Direct AA GDP.PDF all-pass for Full is 32.6, not the vendor-cited 33.2; keep separate. Source checked 12 October 2026; evaluation date unpublished. Transcribed from model-card page 3 image; max effort except Full SWE-Marathon xhigh.

75.3% · raw 75.3 %

Headline · max effort · Own vendor

Source/record date: 2026-10-12

Harness: Meta launch evaluation; 65 professional tasks across 35 occupations; mean rubric reward. Source checked 12 October 2026; evaluation date unpublished. Transcribed from model-card page 3 image; max effort except Full SWE-Marathon xhigh.

78.4% · raw 78.4 %

Headline · max effort · Own vendor

Source/record date: 2026-10-12

Harness: Meta launch evaluation; Updated V2.1 108-task GUI workflows; common internal framework preserving task environments/graders. Opus is Anthropic provider-reported partial reward. Source checked 12 October 2026; evaluation date unpublished. Transcribed from model-card page 3 image; max effort except Full SWE-Marathon xhigh.

39.4% · raw 39.4 %

Headline · max effort · Own vendor

Source/record date: 2026-10-12

Harness: Meta launch evaluation; Strict binary completion; Opus binary result absent, not zero. Source checked 12 October 2026; evaluation date unpublished. Transcribed from model-card page 3 image; max effort except Full SWE-Marathon xhigh.

85.5% · raw 85.5 %

Headline · max effort · Own vendor

Source/record date: 2026-10-12

Harness: Meta launch evaluation; Internal weighted rubric recall; 64K–100K-token multi-turn histories with distractors, conflicts and updates. Source checked 12 October 2026; evaluation date unpublished. Transcribed from model-card page 3 image; max effort except Full SWE-Marathon xhigh.

57.1% · raw 57.1 %

Headline · max effort · Own vendor

Source/record date: 2026-10-12

Harness: Meta launch evaluation; Meta quotes AA reported accuracy; independent exact AA observations remain in evaluators.json. Source checked 12 October 2026; evaluation date unpublished. Transcribed from model-card page 3 image; max effort except Full SWE-Marathon xhigh.

79% · raw 79 %

Headline · max effort · Own vendor

Source/record date: 2026-10-12

Harness: Meta launch evaluation; 113 tasks, 91 repositories, five languages, pass@1. Meta attributes Wonder/Full/Opus to Datacurve; GPT to OpenAI and Fable to Anthropic. Current public Datacurve board predates Wonder; retain vendor citation. Source checked 12 October 2026; evaluation date unpublished. Transcribed from model-card page 3 image; max effort except Full SWE-Marathon xhigh.

57.5% · raw 57.5 %

Headline · xhigh effort · Own vendor

Source/record date: 2026-10-12

Harness: Meta launch evaluation; Abundant AI pass@1; GPT uses Codex and Fable Claude Code, so not a common-harness comparison. Full effort xhigh. Do not conflate with SWE-Marathon v1.1 partial-reward records. Source checked 12 October 2026; evaluation date unpublished. Transcribed from model-card page 3 image; max effort except Full SWE-Marathon xhigh.

59% · raw 59 %

Headline · max effort · Own vendor

Source/record date: 2026-10-12

Harness: Meta launch evaluation; mini-swe-agent, internet disabled after setup, rollouts passing at least 95% of hidden tests; Almost Solved, not fully resolved. Exact task/revision details unpublished. Source checked 12 October 2026; evaluation date unpublished. Transcribed from model-card page 3 image; max effort except Full SWE-Marathon xhigh.

56 · raw 56 index

Headline · max effort · Own vendor

Source/record date: 2026-10-12

Harness: Meta launch evaluation; Rounded vendor citation of AA Intelligence Index v4.3; exact independent index values kept separately. Source checked 12 October 2026; evaluation date unpublished. Transcribed from model-card page 3 image; max effort except Full SWE-Marathon xhigh.

54.2% · raw 54.2 %

Headline · max effort · Own vendor

Source/record date: 2026-10-12

Harness: Meta launch evaluation; Meta internal personal-agent evaluation of persistent work through blockers, multiple interactions and dynamically changing environments, including housing, travel, IT and finance. Source checked 12 October 2026; evaluation date unpublished. Figure/table transcribed visually; do not infer unpublished configuration details.

93.3% · raw 93.3 %

Headline · max effort · Own vendor

Source/record date: 2026-10-12

Harness: Meta launch evaluation; Meta announcement figure: compliance with defensive cyber requests. Higher is better; dataset, denominator, evaluation dates and detailed harness unpublished. Source checked 12 October 2026; evaluation date unpublished. Figure/table transcribed visually; do not infer unpublished configuration details.

96.9% · raw 96.9 %

Headline · max effort · Own vendor

Source/record date: 2026-10-12

Harness: Meta launch evaluation; Meta announcement figure: refusal of cyber misuse requests. Higher is better; dataset, denominator and detailed harness unpublished. Source checked 12 October 2026; evaluation date unpublished. Figure/table transcribed visually; do not infer unpublished configuration details.

98.7% · raw 98.7 %

Headline · max effort · Own vendor

Source/record date: 2026-10-12

Harness: Meta launch evaluation; Meta launch figure BioTIER refusal rate. Language condition unspecified in the chart; retain separately from model-card English/multilingual/adversarial conditions. Source checked 12 October 2026; evaluation date unpublished. Figure/table transcribed visually; do not infer unpublished configuration details.

2.2% · raw 2.2 %

Headline · max effort · Own vendor

Source/record date: 2026-10-12

Harness: Meta launch evaluation; Meta launch figure: rate of mishandling personal information; lower is better. Dataset, denominator and evaluation dates unpublished. Source checked 12 October 2026; evaluation date unpublished. Figure/table transcribed visually; do not infer unpublished configuration details.

10.7% · raw 10.7 %

Headline · max effort · Own vendor

Source/record date: 2026-10-12

Harness: Meta launch evaluation; Meta launch figure: unauthorized code execution attack success rate; lower is better. Exact prompt-injection harness and denominator unpublished. Source checked 12 October 2026; evaluation date unpublished. Figure/table transcribed visually; do not infer unpublished configuration details.

88.4% · raw 88.4 %

Headline · unknown effort · Own vendor

Source/record date: 2026-10-12

Harness: Meta launch evaluation; Model-card page 7: cyber capability; detailed budget/harness not stated. Source checked 12 October 2026; evaluation date unpublished. Figure/table transcribed visually; do not infer unpublished configuration details.

58.9% · raw 58.9 %

Headline · unknown effort · Own vendor

Source/record date: 2026-10-12

Harness: Meta launch evaluation; Page 7; revision and detailed harness unpublished; do not merge with dated V8 subsets. Source checked 12 October 2026; evaluation date unpublished. Figure/table transcribed visually; do not infer unpublished configuration details.

21 flags · raw 21 flags

Headline · unknown effort · Own vendor

Source/record date: 2026-10-12

Harness: Meta launch evaluation; Page 7: intended-vulnerability exploitations, comparable to published ranking. GPT-cyber verification independently confirmed with Gemini 3.1 Pro. Source checked 12 October 2026; evaluation date unpublished. Figure/table transcribed visually; do not infer unpublished configuration details.

92 flags · raw 92 flags

Headline · unknown effort · Own vendor

Source/record date: 2026-10-12

Harness: Meta launch evaluation; Page 7: 92 flags captured, but 71 used unintended vulnerabilities and only 21 intended. Not comparable with intended-only ranking; count, not percent. Source checked 12 October 2026; evaluation date unpublished. Figure/table transcribed visually; do not infer unpublished configuration details.

0.8% · raw 0.8 %

Headline · unknown effort · Own vendor

Source/record date: 2026-10-12

Harness: Meta launch evaluation; Model-card page 8: adversarial refusal attack success rate; lower is better, not refusal rate. Detailed task count and harness unspecified. Source checked 12 October 2026; evaluation date unpublished. Figure/table transcribed visually; do not infer unpublished configuration details.

0.8% · raw 0.8 %

Headline · unknown effort · Own vendor

Source/record date: 2026-10-12

Harness: Meta launch evaluation; Model-card page 8: adversarial refusal attack success rate; lower is better, not refusal rate. Detailed task count and harness unspecified. Source checked 12 October 2026; evaluation date unpublished. Figure/table transcribed visually; do not infer unpublished configuration details.

1.1% · raw 1.1 %

Headline · unknown effort · Own vendor

Source/record date: 2026-10-12

Harness: Meta launch evaluation; Model-card page 8: adversarial refusal attack success rate; lower is better, not refusal rate. PolyGuard cyber subset only. Detailed task count and harness unspecified. Source checked 12 October 2026; evaluation date unpublished. Figure/table transcribed visually; do not infer unpublished configuration details.

2.3% · raw 2.3 %

Headline · unknown effort · Own vendor

Source/record date: 2026-10-12

Harness: Meta launch evaluation; Model-card page 8: adversarial refusal attack success rate; lower is better, not refusal rate. PolyGuard cyber subset only. Detailed task count and harness unspecified. Source checked 12 October 2026; evaluation date unpublished. Figure/table transcribed visually; do not infer unpublished configuration details.

9% · raw 9 %

Headline · unknown effort · Own vendor

Source/record date: 2026-10-12

Harness: Meta launch evaluation; Model-card page 8: adversarial refusal attack success rate; lower is better, not refusal rate. Detailed task count and harness unspecified. Source checked 12 October 2026; evaluation date unpublished. Figure/table transcribed visually; do not infer unpublished configuration details.

8% · raw 8 %

Headline · unknown effort · Own vendor

Source/record date: 2026-10-12

Harness: Meta launch evaluation; Model-card page 8: adversarial refusal attack success rate; lower is better, not refusal rate. Detailed task count and harness unspecified. Source checked 12 October 2026; evaluation date unpublished. Figure/table transcribed visually; do not infer unpublished configuration details.

59.6% · raw 59.6 %

Headline · unknown effort · Own vendor

Source/record date: 2026-10-12

Harness: Meta launch evaluation; Model-card page 9: dual-use proxy capability, not refusal. SeqQA explicitly uses tools; ABC Fragment Design and Screening Evasion have no numeric scores and remain omitted. Meta BDT averages five tasks from three categories. Source checked 12 October 2026; evaluation date unpublished. Figure/table transcribed visually; do not infer unpublished configuration details.

66.5% · raw 66.5 %

Headline · unknown effort · Own vendor

Source/record date: 2026-10-12

Harness: Meta launch evaluation; Model-card page 9: dual-use proxy capability, not refusal. SeqQA explicitly uses tools; ABC Fragment Design and Screening Evasion have no numeric scores and remain omitted. Meta BDT averages five tasks from three categories. Source checked 12 October 2026; evaluation date unpublished. Figure/table transcribed visually; do not infer unpublished configuration details.

55.9% · raw 55.9 %

Headline · unknown effort · Own vendor

Source/record date: 2026-10-12

Harness: Meta launch evaluation; Model-card page 9: dual-use proxy capability, not refusal. SeqQA explicitly uses tools; ABC Fragment Design and Screening Evasion have no numeric scores and remain omitted. Meta BDT averages five tasks from three categories. Source checked 12 October 2026; evaluation date unpublished. Figure/table transcribed visually; do not infer unpublished configuration details.

60.7% · raw 60.7 %

Headline · unknown effort · Own vendor

Source/record date: 2026-10-12

Harness: Meta launch evaluation; Model-card page 9: dual-use proxy capability, not refusal. SeqQA explicitly uses tools; ABC Fragment Design and Screening Evasion have no numeric scores and remain omitted. Meta BDT averages five tasks from three categories. Source checked 12 October 2026; evaluation date unpublished. Figure/table transcribed visually; do not infer unpublished configuration details.

89% · raw 89 %

Headline · unknown effort · Own vendor

Source/record date: 2026-10-12

Harness: Meta launch evaluation; Model-card page 9: dual-use proxy capability, not refusal. SeqQA explicitly uses tools; ABC Fragment Design and Screening Evasion have no numeric scores and remain omitted. Meta BDT averages five tasks from three categories. Source checked 12 October 2026; evaluation date unpublished. Figure/table transcribed visually; do not infer unpublished configuration details.

95.6% · raw 95.6 %

Headline · unknown effort · Own vendor

Source/record date: 2026-10-12

Harness: Meta launch evaluation; Model-card page 9: dual-use proxy capability, not refusal. SeqQA explicitly uses tools; ABC Fragment Design and Screening Evasion have no numeric scores and remain omitted. Meta BDT averages five tasks from three categories. Source checked 12 October 2026; evaluation date unpublished. Figure/table transcribed visually; do not infer unpublished configuration details.

100% · raw 100 %

Headline · unknown effort · Own vendor

Source/record date: 2026-10-12

Harness: Meta launch evaluation; Model-card page 9: dual-use proxy capability, not refusal. SeqQA explicitly uses tools; ABC Fragment Design and Screening Evasion have no numeric scores and remain omitted. Meta BDT averages five tasks from three categories. Source checked 12 October 2026; evaluation date unpublished. Figure/table transcribed visually; do not infer unpublished configuration details.

79.4% · raw 79.4 %

Headline · unknown effort · Own vendor

Source/record date: 2026-10-12

Harness: Meta launch evaluation; Model-card page 9: dual-use proxy capability, not refusal. SeqQA explicitly uses tools; ABC Fragment Design and Screening Evasion have no numeric scores and remain omitted. Meta BDT averages five tasks from three categories. Source checked 12 October 2026; evaluation date unpublished. Figure/table transcribed visually; do not infer unpublished configuration details.

98.7% · raw 98.7 %

Headline · unknown effort · Own vendor

Source/record date: 2026-10-12

Harness: Meta launch evaluation; Model-card page 10: high-risk bio/chem prompt refusal rate, higher is better. Multilingual condition spans ten language buckets. Detailed denominator unpublished. Source checked 12 October 2026; evaluation date unpublished. Figure/table transcribed visually; do not infer unpublished configuration details.

98.7% · raw 98.7 %

Headline · unknown effort · Own vendor

Source/record date: 2026-10-12

Harness: Meta launch evaluation; Model-card page 10: high-risk bio/chem prompt refusal rate, higher is better. Multilingual condition spans ten language buckets. Detailed denominator unpublished. Source checked 12 October 2026; evaluation date unpublished. Figure/table transcribed visually; do not infer unpublished configuration details.

98.2% · raw 98.2 %

Headline · unknown effort · Own vendor

Source/record date: 2026-10-12

Harness: Meta launch evaluation; Model-card page 10: high-risk bio/chem prompt refusal rate, higher is better. Multilingual condition spans ten language buckets. Detailed denominator unpublished. Source checked 12 October 2026; evaluation date unpublished. Figure/table transcribed visually; do not infer unpublished configuration details.

100% · raw 100 %

Headline · unknown effort · Own vendor

Source/record date: 2026-10-12

Harness: Meta launch evaluation; Model-card page 10: high-risk bio/chem prompt refusal rate, higher is better. Multilingual condition spans ten language buckets. Detailed denominator unpublished. Source checked 12 October 2026; evaluation date unpublished. Figure/table transcribed visually; do not infer unpublished configuration details.

100% · raw 100 %

Headline · unknown effort · Own vendor

Source/record date: 2026-10-12

Harness: Meta launch evaluation; Model-card page 10: high-risk bio/chem prompt refusal rate, higher is better. Multilingual condition spans ten language buckets. Detailed denominator unpublished. Source checked 12 October 2026; evaluation date unpublished. Figure/table transcribed visually; do not infer unpublished configuration details.

99.8% · raw 99.8 %

Headline · unknown effort · Own vendor

Source/record date: 2026-10-12

Harness: Meta launch evaluation; Model-card page 10: high-risk bio/chem prompt refusal rate, higher is better. Multilingual condition spans ten language buckets. Detailed denominator unpublished. Source checked 12 October 2026; evaluation date unpublished. Figure/table transcribed visually; do not infer unpublished configuration details.

87.3% · raw 87.3 %

Headline · unknown effort · Own vendor

Source/record date: 2026-10-12

Harness: Meta launch evaluation; Model-card page 10: honesty under pressure; higher is better. Split/harness unspecified, so not merged with public-split MASK records. Source checked 12 October 2026; evaluation date unpublished. Figure/table transcribed visually; do not infer unpublished configuration details.

0% · raw 0 %

Headline · unknown effort · Own vendor

Source/record date: 2026-10-12

Harness: Meta launch evaluation; Model-card page 10: harmful action rate, lower is better; zero is explicitly published, not an imputed gap. Scenario distribution and sample size unpublished. Source checked 12 October 2026; evaluation date unpublished. Figure/table transcribed visually; do not infer unpublished configuration details.

66.787% · raw 66.787 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. GDP-weighted aggregate with revised Finance v2.1, EMB v1.1, HLAB v1.1, Code Migration v1.1 and all-pass Tax. Old Index versions are not comparable. Source default includes provider-refusal fallback tasks for Opus/Sonnet; failure-controlled scores are separate. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":3558.574,"stderr":1.066,"cost_per_test":6.809627,"token_totals":{"input_tokens":11051421288,"output_tokens":96958718,"reasoning_tokens":50663639,"cache_read_tokens":10611361430,"cache_write_tokens":0},"temperature":1.0,"top_p":null,"max_output_tokens":131072,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"Meta","harness":null}

$6.809627/task · raw 6.809627 USD/task

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. GDP-weighted aggregate with revised Finance v2.1, EMB v1.1, HLAB v1.1, Code Migration v1.1 and all-pass Tax. Old Index versions are not comparable. Source default includes provider-refusal fallback tasks for Opus/Sonnet; failure-controlled scores are separate. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":3558.574,"stderr":1.066,"cost_per_test":6.809627,"token_totals":{"input_tokens":11051421288,"output_tokens":96958718,"reasoning_tokens":50663639,"cache_read_tokens":10611361430,"cache_write_tokens":0},"temperature":1.0,"top_p":null,"max_output_tokens":131072,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"Meta","harness":null} Source score standard error describes the benchmark quality score, not uncertainty in cost or latency.

3558.574 s · raw 3558.574 s

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. GDP-weighted aggregate with revised Finance v2.1, EMB v1.1, HLAB v1.1, Code Migration v1.1 and all-pass Tax. Old Index versions are not comparable. Source default includes provider-refusal fallback tasks for Opus/Sonnet; failure-controlled scores are separate. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":3558.574,"stderr":1.066,"cost_per_test":6.809627,"token_totals":{"input_tokens":11051421288,"output_tokens":96958718,"reasoning_tokens":50663639,"cache_read_tokens":10611361430,"cache_write_tokens":0},"temperature":1.0,"top_p":null,"max_output_tokens":131072,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"Meta","harness":null} Source score standard error describes the benchmark quality score, not uncertainty in cost or latency.

85.996% · raw 85.996 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: OpenHands. GDP-weighted aggregate with revised Finance v2.1, EMB v1.1, HLAB v1.1, Code Migration v1.1 and all-pass Tax. Old Index versions are not comparable. Source default includes provider-refusal fallback tasks for Opus/Sonnet; failure-controlled scores are separate. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":648.336,"stderr":1.933,"cost_per_test":2.293448,"token_totals":{"input_tokens":222902077,"output_tokens":2071146,"reasoning_tokens":144351,"cache_read_tokens":215357588,"cache_write_tokens":0},"temperature":1.0,"top_p":null,"max_output_tokens":131072,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"Meta","harness":"OpenHands"}

67.575% · raw 67.575 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. GDP-weighted aggregate with revised Finance v2.1, EMB v1.1, HLAB v1.1, Code Migration v1.1 and all-pass Tax. Old Index versions are not comparable. Source default includes provider-refusal fallback tasks for Opus/Sonnet; failure-controlled scores are separate. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":850.298,"stderr":1.98,"cost_per_test":2.680843,"token_totals":{"input_tokens":389317179,"output_tokens":18193553,"reasoning_tokens":11613004,"cache_read_tokens":188800282,"cache_write_tokens":0},"temperature":1.0,"top_p":null,"max_output_tokens":131072,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"Meta","harness":null}

74.998% · raw 74.998 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. GDP-weighted aggregate with revised Finance v2.1, EMB v1.1, HLAB v1.1, Code Migration v1.1 and all-pass Tax. Old Index versions are not comparable. Source default includes provider-refusal fallback tasks for Opus/Sonnet; failure-controlled scores are separate. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":1684.504,"stderr":2.141,"cost_per_test":4.58223,"token_totals":{"input_tokens":983252171,"output_tokens":8330149,"reasoning_tokens":5099887,"cache_read_tokens":955848482,"cache_write_tokens":0},"temperature":1.0,"top_p":null,"max_output_tokens":131072,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"Meta","harness":null}

61.652% · raw 61.652 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. GDP-weighted aggregate with revised Finance v2.1, EMB v1.1, HLAB v1.1, Code Migration v1.1 and all-pass Tax. Old Index versions are not comparable. Source default includes provider-refusal fallback tasks for Opus/Sonnet; failure-controlled scores are separate. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":15566.738,"stderr":4.371,"cost_per_test":23.236252,"token_totals":{"input_tokens":4997748393,"output_tokens":15267636,"reasoning_tokens":5439846,"cache_read_tokens":4951032316,"cache_write_tokens":0},"temperature":1.0,"top_p":null,"max_output_tokens":131072,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"Meta","harness":null}

53.365% · raw 53.365 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. GDP-weighted aggregate with revised Finance v2.1, EMB v1.1, HLAB v1.1, Code Migration v1.1 and all-pass Tax. Old Index versions are not comparable. Source default includes provider-refusal fallback tasks for Opus/Sonnet; failure-controlled scores are separate. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":1654.378,"stderr":3.467,"cost_per_test":3.595401,"token_totals":{"input_tokens":760777228,"output_tokens":19689738,"reasoning_tokens":13012367,"cache_read_tokens":707647460,"cache_write_tokens":0},"temperature":1.0,"top_p":null,"max_output_tokens":131072,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"Meta","harness":null}

23.333% · raw 23.333 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. GDP-weighted aggregate with revised Finance v2.1, EMB v1.1, HLAB v1.1, Code Migration v1.1 and all-pass Tax. Old Index versions are not comparable. Source default includes provider-refusal fallback tasks for Opus/Sonnet; failure-controlled scores are separate. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":2649.375,"stderr":3.547,"cost_per_test":5.413598,"token_totals":{"input_tokens":1349672470,"output_tokens":13499459,"reasoning_tokens":2307478,"cache_read_tokens":1308495584,"cache_write_tokens":0},"temperature":1.0,"top_p":null,"max_output_tokens":131072,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"Meta","harness":null}

58.081% · raw 58.081 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. GDP-weighted aggregate with revised Finance v2.1, EMB v1.1, HLAB v1.1, Code Migration v1.1 and all-pass Tax. Old Index versions are not comparable. Source default includes provider-refusal fallback tasks for Opus/Sonnet; failure-controlled scores are separate. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":4330.665,"stderr":3.312,"cost_per_test":10.528007,"token_totals":{"input_tokens":2090144775,"output_tokens":9938507,"reasoning_tokens":6960300,"cache_read_tokens":2069605468,"cache_write_tokens":0},"temperature":1.0,"top_p":null,"max_output_tokens":131072,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"Meta","harness":null}

42.997% · raw 42.997 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. GDP-weighted aggregate with revised Finance v2.1, EMB v1.1, HLAB v1.1, Code Migration v1.1 and all-pass Tax. Old Index versions are not comparable. Source default includes provider-refusal fallback tasks for Opus/Sonnet; failure-controlled scores are separate. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":1084.296,"stderr":3.572,"cost_per_test":2.147235,"token_totals":{"input_tokens":257606995,"output_tokens":9968530,"reasoning_tokens":6086406,"cache_read_tokens":214574250,"cache_write_tokens":0},"temperature":1.0,"top_p":null,"max_output_tokens":131072,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"Meta","harness":null}

67.575% · raw 67.575 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. FAB v2 harness on expert-reviewed FABv2.1 questions/rubrics, revised judge jury and hedged-answer prompt. Partial-credit accuracy differs from all-pass. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":850.298,"stderr":1.98,"cost_per_test":2.680843,"token_totals":{"input_tokens":389317179,"output_tokens":18193553,"reasoning_tokens":11613004,"cache_read_tokens":188800282,"cache_write_tokens":0},"temperature":1.0,"top_p":null,"max_output_tokens":131072,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"Meta","harness":null}

$2.680843/task · raw 2.680843 USD/task

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. FAB v2 harness on expert-reviewed FABv2.1 questions/rubrics, revised judge jury and hedged-answer prompt. Partial-credit accuracy differs from all-pass. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":850.298,"stderr":1.98,"cost_per_test":2.680843,"token_totals":{"input_tokens":389317179,"output_tokens":18193553,"reasoning_tokens":11613004,"cache_read_tokens":188800282,"cache_write_tokens":0},"temperature":1.0,"top_p":null,"max_output_tokens":131072,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"Meta","harness":null} Source score standard error describes the benchmark quality score, not uncertainty in cost or latency.

850.298 s · raw 850.298 s

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. FAB v2 harness on expert-reviewed FABv2.1 questions/rubrics, revised judge jury and hedged-answer prompt. Partial-credit accuracy differs from all-pass. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":850.298,"stderr":1.98,"cost_per_test":2.680843,"token_totals":{"input_tokens":389317179,"output_tokens":18193553,"reasoning_tokens":11613004,"cache_read_tokens":188800282,"cache_write_tokens":0},"temperature":1.0,"top_p":null,"max_output_tokens":131072,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"Meta","harness":null} Source score standard error describes the benchmark quality score, not uncertainty in cost or latency.

54.880% · raw 54.88 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. FAB v2 harness on expert-reviewed FABv2.1 questions/rubrics, revised judge jury and hedged-answer prompt. Partial-credit accuracy differs from all-pass. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":850.298,"stderr":2.2,"cost_per_test":2.680843,"token_totals":{"input_tokens":389317179,"output_tokens":18193553,"reasoning_tokens":11613004,"cache_read_tokens":188800282,"cache_write_tokens":0},"temperature":1.0,"top_p":null,"max_output_tokens":131072,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"Meta","harness":null}

76.051% · raw 76.051 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. FAB v2 harness on expert-reviewed FABv2.1 questions/rubrics, revised judge jury and hedged-answer prompt. Partial-credit accuracy differs from all-pass. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":1157.917,"stderr":5.287,"cost_per_test":3.202414,"temperature":1.0,"top_p":null,"max_output_tokens":131072,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"Meta","harness":null}

84.886% · raw 84.886 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. FAB v2 harness on expert-reviewed FABv2.1 questions/rubrics, revised judge jury and hedged-answer prompt. Partial-credit accuracy differs from all-pass. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":683.77,"stderr":4.335,"cost_per_test":2.504633,"temperature":1.0,"top_p":null,"max_output_tokens":131072,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"Meta","harness":null}

67.361% · raw 67.361 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. FAB v2 harness on expert-reviewed FABv2.1 questions/rubrics, revised judge jury and hedged-answer prompt. Partial-credit accuracy differs from all-pass. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":520.944,"stderr":6.694,"cost_per_test":1.430893,"temperature":1.0,"top_p":null,"max_output_tokens":131072,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"Meta","harness":null}

55.011% · raw 55.011 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. FAB v2 harness on expert-reviewed FABv2.1 questions/rubrics, revised judge jury and hedged-answer prompt. Partial-credit accuracy differs from all-pass. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":1094.974,"stderr":6.639,"cost_per_test":3.896655,"temperature":1.0,"top_p":null,"max_output_tokens":131072,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"Meta","harness":null}

44.181% · raw 44.181 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. FAB v2 harness on expert-reviewed FABv2.1 questions/rubrics, revised judge jury and hedged-answer prompt. Partial-credit accuracy differs from all-pass. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":1253.085,"stderr":7.015,"cost_per_test":4.151451,"temperature":1.0,"top_p":null,"max_output_tokens":131072,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"Meta","harness":null}

74.737% · raw 74.737 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. FAB v2 harness on expert-reviewed FABv2.1 questions/rubrics, revised judge jury and hedged-answer prompt. Partial-credit accuracy differs from all-pass. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":888.018,"stderr":6.048,"cost_per_test":2.686259,"temperature":1.0,"top_p":null,"max_output_tokens":131072,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"Meta","harness":null}

84.794% · raw 84.794 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. FAB v2 harness on expert-reviewed FABv2.1 questions/rubrics, revised judge jury and hedged-answer prompt. Partial-credit accuracy differs from all-pass. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":436.722,"stderr":4.939,"cost_per_test":1.018528,"temperature":1.0,"top_p":null,"max_output_tokens":131072,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"Meta","harness":null}

80.595% · raw 80.595 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. FAB v2 harness on expert-reviewed FABv2.1 questions/rubrics, revised judge jury and hedged-answer prompt. Partial-credit accuracy differs from all-pass. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":962.739,"stderr":5.226,"cost_per_test":3.162266,"temperature":1.0,"top_p":null,"max_output_tokens":131072,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"Meta","harness":null}

40.557% · raw 40.557 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. FAB v2 harness on expert-reviewed FABv2.1 questions/rubrics, revised judge jury and hedged-answer prompt. Partial-credit accuracy differs from all-pass. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":735.548,"stderr":6.661,"cost_per_test":2.261304,"temperature":1.0,"top_p":null,"max_output_tokens":131072,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"Meta","harness":null}

74.998% · raw 74.998 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Seven equally weighted category scores combining Template and Scratch. GPT-6 Luna Scratch judge and Template sheet-name scoring fixes; not comparable with v1.0. Partial credit is not client-ready quality. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":1684.504,"stderr":2.141,"cost_per_test":4.58223,"token_totals":{"input_tokens":983252171,"output_tokens":8330149,"reasoning_tokens":5099887,"cache_read_tokens":955848482,"cache_write_tokens":0},"temperature":1.0,"top_p":null,"max_output_tokens":131072,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"Meta","harness":null}

$4.58223/task · raw 4.58223 USD/task

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Seven equally weighted category scores combining Template and Scratch. GPT-6 Luna Scratch judge and Template sheet-name scoring fixes; not comparable with v1.0. Partial credit is not client-ready quality. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":1684.504,"stderr":2.141,"cost_per_test":4.58223,"token_totals":{"input_tokens":983252171,"output_tokens":8330149,"reasoning_tokens":5099887,"cache_read_tokens":955848482,"cache_write_tokens":0},"temperature":1.0,"top_p":null,"max_output_tokens":131072,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"Meta","harness":null} Source score standard error describes the benchmark quality score, not uncertainty in cost or latency.

1684.504 s · raw 1684.504 s

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Seven equally weighted category scores combining Template and Scratch. GPT-6 Luna Scratch judge and Template sheet-name scoring fixes; not comparable with v1.0. Partial credit is not client-ready quality. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":1684.504,"stderr":2.141,"cost_per_test":4.58223,"token_totals":{"input_tokens":983252171,"output_tokens":8330149,"reasoning_tokens":5099887,"cache_read_tokens":955848482,"cache_write_tokens":0},"temperature":1.0,"top_p":null,"max_output_tokens":131072,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"Meta","harness":null} Source score standard error describes the benchmark quality score, not uncertainty in cost or latency.

79.861% · raw 79.861 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Seven equally weighted category scores combining Template and Scratch. GPT-6 Luna Scratch judge and Template sheet-name scoring fixes; not comparable with v1.0. Partial credit is not client-ready quality. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":1834.408,"stderr":2.635,"cost_per_test":5.314337,"temperature":1.0,"top_p":null,"max_output_tokens":131072,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"Meta","harness":null}

70.032% · raw 70.032 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Seven equally weighted category scores combining Template and Scratch. GPT-6 Luna Scratch judge and Template sheet-name scoring fixes; not comparable with v1.0. Partial credit is not client-ready quality. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":1534.599,"stderr":2.614,"cost_per_test":3.850123,"temperature":1.0,"top_p":null,"max_output_tokens":131072,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"Meta","harness":null}

63.496% · raw 63.496 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Seven equally weighted category scores combining Template and Scratch. GPT-6 Luna Scratch judge and Template sheet-name scoring fixes; not comparable with v1.0. Partial credit is not client-ready quality. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":2261.66,"stderr":4.727,"cost_per_test":7.053456,"temperature":1.0,"top_p":null,"max_output_tokens":131072,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"Meta","harness":null}

65.743% · raw 65.743 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Seven equally weighted category scores combining Template and Scratch. GPT-6 Luna Scratch judge and Template sheet-name scoring fixes; not comparable with v1.0. Partial credit is not client-ready quality. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":1613.553,"stderr":7.305,"cost_per_test":3.903815,"temperature":1.0,"top_p":null,"max_output_tokens":131072,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"Meta","harness":null}

83.423% · raw 83.423 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Seven equally weighted category scores combining Template and Scratch. GPT-6 Luna Scratch judge and Template sheet-name scoring fixes; not comparable with v1.0. Partial credit is not client-ready quality. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":1362.463,"stderr":3.078,"cost_per_test":3.075349,"temperature":1.0,"top_p":null,"max_output_tokens":131072,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"Meta","harness":null}

74.631% · raw 74.631 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Seven equally weighted category scores combining Template and Scratch. GPT-6 Luna Scratch judge and Template sheet-name scoring fixes; not comparable with v1.0. Partial credit is not client-ready quality. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":1570.029,"stderr":5.192,"cost_per_test":3.836765,"temperature":1.0,"top_p":null,"max_output_tokens":131072,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"Meta","harness":null}

80.197% · raw 80.197 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Seven equally weighted category scores combining Template and Scratch. GPT-6 Luna Scratch judge and Template sheet-name scoring fixes; not comparable with v1.0. Partial credit is not client-ready quality. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":1974.807,"stderr":3.914,"cost_per_test":6.137566,"temperature":1.0,"top_p":null,"max_output_tokens":131072,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"Meta","harness":null}

86.553% · raw 86.553 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Seven equally weighted category scores combining Template and Scratch. GPT-6 Luna Scratch judge and Template sheet-name scoring fixes; not comparable with v1.0. Partial credit is not client-ready quality. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":1361.668,"stderr":3.59,"cost_per_test":3.621869,"temperature":1.0,"top_p":null,"max_output_tokens":131072,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"Meta","harness":null}

70.941% · raw 70.941 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Seven equally weighted category scores combining Template and Scratch. GPT-6 Luna Scratch judge and Template sheet-name scoring fixes; not comparable with v1.0. Partial credit is not client-ready quality. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":1515.655,"stderr":3.367,"cost_per_test":3.877366,"temperature":1.0,"top_p":null,"max_output_tokens":131072,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"Meta","harness":null}

70.179% · raw 70.179 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Seven equally weighted category scores combining Template and Scratch. GPT-6 Luna Scratch judge and Template sheet-name scoring fixes; not comparable with v1.0. Partial credit is not client-ready quality. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":1684.504,"stderr":2.141,"cost_per_test":4.58223,"token_totals":{"input_tokens":983252171,"output_tokens":8330149,"reasoning_tokens":5099887,"cache_read_tokens":955848482,"cache_write_tokens":0},"temperature":1.0,"top_p":null,"max_output_tokens":131072,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"Meta","harness":null}

56.814% · raw 56.814 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Seven equally weighted category scores combining Template and Scratch. GPT-6 Luna Scratch judge and Template sheet-name scoring fixes; not comparable with v1.0. Partial credit is not client-ready quality. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":1684.504,"stderr":2.141,"cost_per_test":4.58223,"token_totals":{"input_tokens":983252171,"output_tokens":8330149,"reasoning_tokens":5099887,"cache_read_tokens":955848482,"cache_write_tokens":0},"temperature":1.0,"top_p":null,"max_output_tokens":131072,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"Meta","harness":null}

71.220% · raw 71.22 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Seven equally weighted category scores combining Template and Scratch. GPT-6 Luna Scratch judge and Template sheet-name scoring fixes; not comparable with v1.0. Partial credit is not client-ready quality. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":1684.504,"stderr":2.141,"cost_per_test":4.58223,"token_totals":{"input_tokens":983252171,"output_tokens":8330149,"reasoning_tokens":5099887,"cache_read_tokens":955848482,"cache_write_tokens":0},"temperature":1.0,"top_p":null,"max_output_tokens":131072,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"Meta","harness":null}

60.265% · raw 60.265 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Seven equally weighted category scores combining Template and Scratch. GPT-6 Luna Scratch judge and Template sheet-name scoring fixes; not comparable with v1.0. Partial credit is not client-ready quality. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":1684.504,"stderr":2.141,"cost_per_test":4.58223,"token_totals":{"input_tokens":983252171,"output_tokens":8330149,"reasoning_tokens":5099887,"cache_read_tokens":955848482,"cache_write_tokens":0},"temperature":1.0,"top_p":null,"max_output_tokens":131072,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"Meta","harness":null}

87.451% · raw 87.451 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Seven equally weighted category scores combining Template and Scratch. GPT-6 Luna Scratch judge and Template sheet-name scoring fixes; not comparable with v1.0. Partial credit is not client-ready quality. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":1684.504,"stderr":2.141,"cost_per_test":4.58223,"token_totals":{"input_tokens":983252171,"output_tokens":8330149,"reasoning_tokens":5099887,"cache_read_tokens":955848482,"cache_write_tokens":0},"temperature":1.0,"top_p":null,"max_output_tokens":131072,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"Meta","harness":null}

79.396% · raw 79.396 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Seven equally weighted category scores combining Template and Scratch. GPT-6 Luna Scratch judge and Template sheet-name scoring fixes; not comparable with v1.0. Partial credit is not client-ready quality. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":1684.504,"stderr":2.141,"cost_per_test":4.58223,"token_totals":{"input_tokens":983252171,"output_tokens":8330149,"reasoning_tokens":5099887,"cache_read_tokens":955848482,"cache_write_tokens":0},"temperature":1.0,"top_p":null,"max_output_tokens":131072,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"Meta","harness":null}

81.012% · raw 81.012 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Seven equally weighted category scores combining Template and Scratch. GPT-6 Luna Scratch judge and Template sheet-name scoring fixes; not comparable with v1.0. Partial credit is not client-ready quality. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":1684.504,"stderr":2.141,"cost_per_test":4.58223,"token_totals":{"input_tokens":983252171,"output_tokens":8330149,"reasoning_tokens":5099887,"cache_read_tokens":955848482,"cache_write_tokens":0},"temperature":1.0,"top_p":null,"max_output_tokens":131072,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"Meta","harness":null}

68.250% · raw 68.25 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Seven equally weighted category scores combining Template and Scratch. GPT-6 Luna Scratch judge and Template sheet-name scoring fixes; not comparable with v1.0. Partial credit is not client-ready quality. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":1684.504,"stderr":2.141,"cost_per_test":4.58223,"token_totals":{"input_tokens":983252171,"output_tokens":8330149,"reasoning_tokens":5099887,"cache_read_tokens":955848482,"cache_write_tokens":0},"temperature":1.0,"top_p":null,"max_output_tokens":131072,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"Meta","harness":null}

87.495% · raw 87.495 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Seven equally weighted category scores combining Template and Scratch. GPT-6 Luna Scratch judge and Template sheet-name scoring fixes; not comparable with v1.0. Partial credit is not client-ready quality. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":1684.504,"stderr":2.141,"cost_per_test":4.58223,"token_totals":{"input_tokens":983252171,"output_tokens":8330149,"reasoning_tokens":5099887,"cache_read_tokens":955848482,"cache_write_tokens":0},"temperature":1.0,"top_p":null,"max_output_tokens":131072,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"Meta","harness":null}

72.900% · raw 72.9 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Seven equally weighted category scores combining Template and Scratch. GPT-6 Luna Scratch judge and Template sheet-name scoring fixes; not comparable with v1.0. Partial credit is not client-ready quality. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":1684.504,"stderr":2.141,"cost_per_test":4.58223,"token_totals":{"input_tokens":983252171,"output_tokens":8330149,"reasoning_tokens":5099887,"cache_read_tokens":955848482,"cache_write_tokens":0},"temperature":1.0,"top_p":null,"max_output_tokens":131072,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"Meta","harness":null}

89.005% · raw 89.005 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Seven equally weighted category scores combining Template and Scratch. GPT-6 Luna Scratch judge and Template sheet-name scoring fixes; not comparable with v1.0. Partial credit is not client-ready quality. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":1684.504,"stderr":2.141,"cost_per_test":4.58223,"token_totals":{"input_tokens":983252171,"output_tokens":8330149,"reasoning_tokens":5099887,"cache_read_tokens":955848482,"cache_write_tokens":0},"temperature":1.0,"top_p":null,"max_output_tokens":131072,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"Meta","harness":null}

84.100% · raw 84.1 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Seven equally weighted category scores combining Template and Scratch. GPT-6 Luna Scratch judge and Template sheet-name scoring fixes; not comparable with v1.0. Partial credit is not client-ready quality. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":1684.504,"stderr":2.141,"cost_per_test":4.58223,"token_totals":{"input_tokens":983252171,"output_tokens":8330149,"reasoning_tokens":5099887,"cache_read_tokens":955848482,"cache_write_tokens":0},"temperature":1.0,"top_p":null,"max_output_tokens":131072,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"Meta","harness":null}

71.617% · raw 71.617 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Seven equally weighted category scores combining Template and Scratch. GPT-6 Luna Scratch judge and Template sheet-name scoring fixes; not comparable with v1.0. Partial credit is not client-ready quality. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":1684.504,"stderr":2.141,"cost_per_test":4.58223,"token_totals":{"input_tokens":983252171,"output_tokens":8330149,"reasoning_tokens":5099887,"cache_read_tokens":955848482,"cache_write_tokens":0},"temperature":1.0,"top_p":null,"max_output_tokens":131072,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"Meta","harness":null}

70.265% · raw 70.265 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Seven equally weighted category scores combining Template and Scratch. GPT-6 Luna Scratch judge and Template sheet-name scoring fixes; not comparable with v1.0. Partial credit is not client-ready quality. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":1684.504,"stderr":2.141,"cost_per_test":4.58223,"token_totals":{"input_tokens":983252171,"output_tokens":8330149,"reasoning_tokens":5099887,"cache_read_tokens":955848482,"cache_write_tokens":0},"temperature":1.0,"top_p":null,"max_output_tokens":131072,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"Meta","harness":null}

61.835% · raw 61.835 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Seven equally weighted category scores combining Template and Scratch. GPT-6 Luna Scratch judge and Template sheet-name scoring fixes; not comparable with v1.0. Partial credit is not client-ready quality. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":1684.504,"stderr":2.141,"cost_per_test":4.58223,"token_totals":{"input_tokens":983252171,"output_tokens":8330149,"reasoning_tokens":5099887,"cache_read_tokens":955848482,"cache_write_tokens":0},"temperature":1.0,"top_p":null,"max_output_tokens":131072,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"Meta","harness":null}

88.417% · raw 88.417 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Seven equally weighted category scores combining Template and Scratch. GPT-6 Luna Scratch judge and Template sheet-name scoring fixes; not comparable with v1.0. Partial credit is not client-ready quality. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":1684.504,"stderr":2.141,"cost_per_test":4.58223,"token_totals":{"input_tokens":983252171,"output_tokens":8330149,"reasoning_tokens":5099887,"cache_read_tokens":955848482,"cache_write_tokens":0},"temperature":1.0,"top_p":null,"max_output_tokens":131072,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"Meta","harness":null}

79.753% · raw 79.753 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Seven equally weighted category scores combining Template and Scratch. GPT-6 Luna Scratch judge and Template sheet-name scoring fixes; not comparable with v1.0. Partial credit is not client-ready quality. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":1684.504,"stderr":2.141,"cost_per_test":4.58223,"token_totals":{"input_tokens":983252171,"output_tokens":8330149,"reasoning_tokens":5099887,"cache_read_tokens":955848482,"cache_write_tokens":0},"temperature":1.0,"top_p":null,"max_output_tokens":131072,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"Meta","harness":null}

62.790% · raw 62.79 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: Pi. Pi agent harness, revised hidden tests, full 120 CLI + 10 COBOL tasks; 75% CLI and 25% COBOL weighting. Distinct from Index selected 50 CLI +10 COBOL subset and old mini-SWE-agent v1. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":15501.972,"stderr":3.175,"cost_per_test":23.584618,"token_totals":{"input_tokens":11830929459,"output_tokens":35940066,"reasoning_tokens":12452527,"cache_read_tokens":11728671116,"cache_write_tokens":0},"temperature":1.0,"top_p":null,"max_output_tokens":131072,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"Meta","harness":null}

$23.584618/task · raw 23.584618 USD/task

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: Pi. Pi agent harness, revised hidden tests, full 120 CLI + 10 COBOL tasks; 75% CLI and 25% COBOL weighting. Distinct from Index selected 50 CLI +10 COBOL subset and old mini-SWE-agent v1. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":15501.972,"stderr":3.175,"cost_per_test":23.584618,"token_totals":{"input_tokens":11830929459,"output_tokens":35940066,"reasoning_tokens":12452527,"cache_read_tokens":11728671116,"cache_write_tokens":0},"temperature":1.0,"top_p":null,"max_output_tokens":131072,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"Meta","harness":null} Source score standard error describes the benchmark quality score, not uncertainty in cost or latency.

15501.972 s · raw 15501.972 s

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: Pi. Pi agent harness, revised hidden tests, full 120 CLI + 10 COBOL tasks; 75% CLI and 25% COBOL weighting. Distinct from Index selected 50 CLI +10 COBOL subset and old mini-SWE-agent v1. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":15501.972,"stderr":3.175,"cost_per_test":23.584618,"token_totals":{"input_tokens":11830929459,"output_tokens":35940066,"reasoning_tokens":12452527,"cache_read_tokens":11728671116,"cache_write_tokens":0},"temperature":1.0,"top_p":null,"max_output_tokens":131072,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"Meta","harness":null} Source score standard error describes the benchmark quality score, not uncertainty in cost or latency.

52.721% · raw 52.721 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: Pi. Pi agent harness, revised hidden tests, full 120 CLI + 10 COBOL tasks; 75% CLI and 25% COBOL weighting. Distinct from Index selected 50 CLI +10 COBOL subset and old mini-SWE-agent v1. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":17699.003,"stderr":3.532,"cost_per_test":28.113158,"temperature":1.0,"top_p":null,"max_output_tokens":131072,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"Meta","harness":null}

93.000% · raw 93 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: Pi. Pi agent harness, revised hidden tests, full 120 CLI + 10 COBOL tasks; 75% CLI and 25% COBOL weighting. Distinct from Index selected 50 CLI +10 COBOL subset and old mini-SWE-agent v1. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":8910.877,"stderr":7.0,"cost_per_test":9.999,"temperature":1.0,"top_p":null,"max_output_tokens":131072,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"Meta","harness":null}

3.238 points/5 · raw 3.238 points/5

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: Pi. Pi agent harness, revised hidden tests, full 120 CLI + 10 COBOL tasks; 75% CLI and 25% COBOL weighting. Distinct from Index selected 50 CLI +10 COBOL subset and old mini-SWE-agent v1. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":15501.972,"stderr":0.043,"cost_per_test":23.584618,"temperature":1.0,"top_p":null,"max_output_tokens":131072,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"Meta","harness":null}

23.333% · raw 23.333 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Harvey held-out legal work protocol, internet disabled, GPT5.5 medium and Sonnet4.6 judges. Task pass requires every criterion; final is mean of two judges. Criteria-pass rate is separate. DOCX tracked-change grading fix. Distinct from AA hallucination-gated LAB. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":2649.375,"stderr":3.547,"cost_per_test":5.413598,"token_totals":{"input_tokens":1349672470,"output_tokens":13499459,"reasoning_tokens":2307478,"cache_read_tokens":1308495584,"cache_write_tokens":0},"temperature":1.0,"top_p":null,"max_output_tokens":131072,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"Meta","harness":null}

$5.413598/task · raw 5.413598 USD/task

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Harvey held-out legal work protocol, internet disabled, GPT5.5 medium and Sonnet4.6 judges. Task pass requires every criterion; final is mean of two judges. Criteria-pass rate is separate. DOCX tracked-change grading fix. Distinct from AA hallucination-gated LAB. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":2649.375,"stderr":3.547,"cost_per_test":5.413598,"token_totals":{"input_tokens":1349672470,"output_tokens":13499459,"reasoning_tokens":2307478,"cache_read_tokens":1308495584,"cache_write_tokens":0},"temperature":1.0,"top_p":null,"max_output_tokens":131072,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"Meta","harness":null} Source score standard error describes the benchmark quality score, not uncertainty in cost or latency.

2649.375 s · raw 2649.375 s

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Harvey held-out legal work protocol, internet disabled, GPT5.5 medium and Sonnet4.6 judges. Task pass requires every criterion; final is mean of two judges. Criteria-pass rate is separate. DOCX tracked-change grading fix. Distinct from AA hallucination-gated LAB. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":2649.375,"stderr":3.547,"cost_per_test":5.413598,"token_totals":{"input_tokens":1349672470,"output_tokens":13499459,"reasoning_tokens":2307478,"cache_read_tokens":1308495584,"cache_write_tokens":0},"temperature":1.0,"top_p":null,"max_output_tokens":131072,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"Meta","harness":null} Source score standard error describes the benchmark quality score, not uncertainty in cost or latency.

40.000% · raw 40 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Harvey held-out legal work protocol, internet disabled, GPT5.5 medium and Sonnet4.6 judges. Task pass requires every criterion; final is mean of two judges. Criteria-pass rate is separate. DOCX tracked-change grading fix. Distinct from AA hallucination-gated LAB. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":2044.898,"stderr":18.708,"cost_per_test":6.005504,"temperature":1.0,"top_p":null,"max_output_tokens":131072,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"Meta","harness":null}

10.000% · raw 10 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Harvey held-out legal work protocol, internet disabled, GPT5.5 medium and Sonnet4.6 judges. Task pass requires every criterion; final is mean of two judges. Criteria-pass rate is separate. DOCX tracked-change grading fix. Distinct from AA hallucination-gated LAB. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":2620.15,"stderr":10.0,"cost_per_test":5.114031,"temperature":1.0,"top_p":null,"max_output_tokens":131072,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"Meta","harness":null}

30.000% · raw 30 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Harvey held-out legal work protocol, internet disabled, GPT5.5 medium and Sonnet4.6 judges. Task pass requires every criterion; final is mean of two judges. Criteria-pass rate is separate. DOCX tracked-change grading fix. Distinct from AA hallucination-gated LAB. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":3124.788,"stderr":20.0,"cost_per_test":5.41453,"temperature":1.0,"top_p":null,"max_output_tokens":131072,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"Meta","harness":null}

30.000% · raw 30 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Harvey held-out legal work protocol, internet disabled, GPT5.5 medium and Sonnet4.6 judges. Task pass requires every criterion; final is mean of two judges. Criteria-pass rate is separate. DOCX tracked-change grading fix. Distinct from AA hallucination-gated LAB. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":2901.7,"stderr":20.0,"cost_per_test":6.272163,"temperature":1.0,"top_p":null,"max_output_tokens":131072,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"Meta","harness":null}

20.000% · raw 20 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Harvey held-out legal work protocol, internet disabled, GPT5.5 medium and Sonnet4.6 judges. Task pass requires every criterion; final is mean of two judges. Criteria-pass rate is separate. DOCX tracked-change grading fix. Distinct from AA hallucination-gated LAB. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":2353.468,"stderr":20.0,"cost_per_test":4.235198,"temperature":1.0,"top_p":null,"max_output_tokens":131072,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"Meta","harness":null}

20.000% · raw 20 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Harvey held-out legal work protocol, internet disabled, GPT5.5 medium and Sonnet4.6 judges. Task pass requires every criterion; final is mean of two judges. Criteria-pass rate is separate. DOCX tracked-change grading fix. Distinct from AA hallucination-gated LAB. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":2735.374,"stderr":20.0,"cost_per_test":5.218937,"temperature":1.0,"top_p":null,"max_output_tokens":131072,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"Meta","harness":null}

30.000% · raw 30 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Harvey held-out legal work protocol, internet disabled, GPT5.5 medium and Sonnet4.6 judges. Task pass requires every criterion; final is mean of two judges. Criteria-pass rate is separate. DOCX tracked-change grading fix. Distinct from AA hallucination-gated LAB. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":2894.524,"stderr":20.0,"cost_per_test":5.165325,"temperature":1.0,"top_p":null,"max_output_tokens":131072,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"Meta","harness":null}

20.000% · raw 20 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Harvey held-out legal work protocol, internet disabled, GPT5.5 medium and Sonnet4.6 judges. Task pass requires every criterion; final is mean of two judges. Criteria-pass rate is separate. DOCX tracked-change grading fix. Distinct from AA hallucination-gated LAB. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":2553.994,"stderr":20.0,"cost_per_test":5.724329,"temperature":1.0,"top_p":null,"max_output_tokens":131072,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"Meta","harness":null}

0.000% · raw 0 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Harvey held-out legal work protocol, internet disabled, GPT5.5 medium and Sonnet4.6 judges. Task pass requires every criterion; final is mean of two judges. Criteria-pass rate is separate. DOCX tracked-change grading fix. Distinct from AA hallucination-gated LAB. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":2502.37,"stderr":0.0,"cost_per_test":6.394651,"temperature":1.0,"top_p":null,"max_output_tokens":131072,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"Meta","harness":null}

0.000% · raw 0 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Harvey held-out legal work protocol, internet disabled, GPT5.5 medium and Sonnet4.6 judges. Task pass requires every criterion; final is mean of two judges. Criteria-pass rate is separate. DOCX tracked-change grading fix. Distinct from AA hallucination-gated LAB. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":2510.782,"stderr":0.0,"cost_per_test":4.65707,"temperature":1.0,"top_p":null,"max_output_tokens":131072,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"Meta","harness":null}

60.000% · raw 60 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Harvey held-out legal work protocol, internet disabled, GPT5.5 medium and Sonnet4.6 judges. Task pass requires every criterion; final is mean of two judges. Criteria-pass rate is separate. DOCX tracked-change grading fix. Distinct from AA hallucination-gated LAB. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":3120.71,"stderr":24.495,"cost_per_test":6.264605,"temperature":1.0,"top_p":null,"max_output_tokens":131072,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"Meta","harness":null}

20.000% · raw 20 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Harvey held-out legal work protocol, internet disabled, GPT5.5 medium and Sonnet4.6 judges. Task pass requires every criterion; final is mean of two judges. Criteria-pass rate is separate. DOCX tracked-change grading fix. Distinct from AA hallucination-gated LAB. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":2908.808,"stderr":20.0,"cost_per_test":4.77579,"temperature":1.0,"top_p":null,"max_output_tokens":131072,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"Meta","harness":null}

40.000% · raw 40 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Harvey held-out legal work protocol, internet disabled, GPT5.5 medium and Sonnet4.6 judges. Task pass requires every criterion; final is mean of two judges. Criteria-pass rate is separate. DOCX tracked-change grading fix. Distinct from AA hallucination-gated LAB. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":3311.582,"stderr":24.495,"cost_per_test":6.091472,"temperature":1.0,"top_p":null,"max_output_tokens":131072,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"Meta","harness":null}

40.000% · raw 40 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Harvey held-out legal work protocol, internet disabled, GPT5.5 medium and Sonnet4.6 judges. Task pass requires every criterion; final is mean of two judges. Criteria-pass rate is separate. DOCX tracked-change grading fix. Distinct from AA hallucination-gated LAB. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":2539.434,"stderr":18.708,"cost_per_test":4.848495,"temperature":1.0,"top_p":null,"max_output_tokens":131072,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"Meta","harness":null}

20.000% · raw 20 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Harvey held-out legal work protocol, internet disabled, GPT5.5 medium and Sonnet4.6 judges. Task pass requires every criterion; final is mean of two judges. Criteria-pass rate is separate. DOCX tracked-change grading fix. Distinct from AA hallucination-gated LAB. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":2604.304,"stderr":20.0,"cost_per_test":4.272067,"temperature":1.0,"top_p":null,"max_output_tokens":131072,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"Meta","harness":null}

30.000% · raw 30 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Harvey held-out legal work protocol, internet disabled, GPT5.5 medium and Sonnet4.6 judges. Task pass requires every criterion; final is mean of two judges. Criteria-pass rate is separate. DOCX tracked-change grading fix. Distinct from AA hallucination-gated LAB. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":2466.874,"stderr":20.0,"cost_per_test":4.928353,"temperature":1.0,"top_p":null,"max_output_tokens":131072,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"Meta","harness":null}

30.000% · raw 30 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Harvey held-out legal work protocol, internet disabled, GPT5.5 medium and Sonnet4.6 judges. Task pass requires every criterion; final is mean of two judges. Criteria-pass rate is separate. DOCX tracked-change grading fix. Distinct from AA hallucination-gated LAB. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":2474.152,"stderr":20.0,"cost_per_test":5.894615,"temperature":1.0,"top_p":null,"max_output_tokens":131072,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"Meta","harness":null}

20.000% · raw 20 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Harvey held-out legal work protocol, internet disabled, GPT5.5 medium and Sonnet4.6 judges. Task pass requires every criterion; final is mean of two judges. Criteria-pass rate is separate. DOCX tracked-change grading fix. Distinct from AA hallucination-gated LAB. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":2063.232,"stderr":12.247,"cost_per_test":4.26355,"temperature":1.0,"top_p":null,"max_output_tokens":131072,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"Meta","harness":null}

0.000% · raw 0 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Harvey held-out legal work protocol, internet disabled, GPT5.5 medium and Sonnet4.6 judges. Task pass requires every criterion; final is mean of two judges. Criteria-pass rate is separate. DOCX tracked-change grading fix. Distinct from AA hallucination-gated LAB. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":2486.446,"stderr":0.0,"cost_per_test":5.721406,"temperature":1.0,"top_p":null,"max_output_tokens":131072,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"Meta","harness":null}

40.000% · raw 40 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Harvey held-out legal work protocol, internet disabled, GPT5.5 medium and Sonnet4.6 judges. Task pass requires every criterion; final is mean of two judges. Criteria-pass rate is separate. DOCX tracked-change grading fix. Distinct from AA hallucination-gated LAB. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":2871.286,"stderr":24.495,"cost_per_test":6.578616,"temperature":1.0,"top_p":null,"max_output_tokens":131072,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"Meta","harness":null}

20.000% · raw 20 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Harvey held-out legal work protocol, internet disabled, GPT5.5 medium and Sonnet4.6 judges. Task pass requires every criterion; final is mean of two judges. Criteria-pass rate is separate. DOCX tracked-change grading fix. Distinct from AA hallucination-gated LAB. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":2426.824,"stderr":20.0,"cost_per_test":6.8176,"temperature":1.0,"top_p":null,"max_output_tokens":131072,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"Meta","harness":null}

0.000% · raw 0 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Harvey held-out legal work protocol, internet disabled, GPT5.5 medium and Sonnet4.6 judges. Task pass requires every criterion; final is mean of two judges. Criteria-pass rate is separate. DOCX tracked-change grading fix. Distinct from AA hallucination-gated LAB. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":2475.476,"stderr":0.0,"cost_per_test":5.723511,"temperature":1.0,"top_p":null,"max_output_tokens":131072,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"Meta","harness":null}

40.000% · raw 40 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Harvey held-out legal work protocol, internet disabled, GPT5.5 medium and Sonnet4.6 judges. Task pass requires every criterion; final is mean of two judges. Criteria-pass rate is separate. DOCX tracked-change grading fix. Distinct from AA hallucination-gated LAB. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":2998.392,"stderr":18.708,"cost_per_test":4.090553,"temperature":1.0,"top_p":null,"max_output_tokens":131072,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"Meta","harness":null}

0.000% · raw 0 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Harvey held-out legal work protocol, internet disabled, GPT5.5 medium and Sonnet4.6 judges. Task pass requires every criterion; final is mean of two judges. Criteria-pass rate is separate. DOCX tracked-change grading fix. Distinct from AA hallucination-gated LAB. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":2595.43,"stderr":0.0,"cost_per_test":5.453971,"temperature":1.0,"top_p":null,"max_output_tokens":131072,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"Meta","harness":null}

96.126% · raw 96.126 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Harvey held-out legal work protocol, internet disabled, GPT5.5 medium and Sonnet4.6 judges. Task pass requires every criterion; final is mean of two judges. Criteria-pass rate is separate. DOCX tracked-change grading fix. Distinct from AA hallucination-gated LAB. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":2649.375,"stderr":0.232,"cost_per_test":5.413598,"token_totals":{"input_tokens":1349672470,"output_tokens":13499459,"reasoning_tokens":2307478,"cache_read_tokens":1308495584,"cache_write_tokens":0},"temperature":1.0,"top_p":null,"max_output_tokens":131072,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"Meta","harness":null}

96.920% · raw 96.92 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Harvey held-out legal work protocol, internet disabled, GPT5.5 medium and Sonnet4.6 judges. Task pass requires every criterion; final is mean of two judges. Criteria-pass rate is separate. DOCX tracked-change grading fix. Distinct from AA hallucination-gated LAB. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":2044.898,"stderr":1.042,"cost_per_test":6.005504,"temperature":1.0,"top_p":null,"max_output_tokens":131072,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"Meta","harness":null}

96.595% · raw 96.595 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Harvey held-out legal work protocol, internet disabled, GPT5.5 medium and Sonnet4.6 judges. Task pass requires every criterion; final is mean of two judges. Criteria-pass rate is separate. DOCX tracked-change grading fix. Distinct from AA hallucination-gated LAB. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":2620.15,"stderr":1.088,"cost_per_test":5.114031,"temperature":1.0,"top_p":null,"max_output_tokens":131072,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"Meta","harness":null}

97.575% · raw 97.575 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Harvey held-out legal work protocol, internet disabled, GPT5.5 medium and Sonnet4.6 judges. Task pass requires every criterion; final is mean of two judges. Criteria-pass rate is separate. DOCX tracked-change grading fix. Distinct from AA hallucination-gated LAB. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":3124.788,"stderr":0.941,"cost_per_test":5.41453,"temperature":1.0,"top_p":null,"max_output_tokens":131072,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"Meta","harness":null}

94.983% · raw 94.983 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Harvey held-out legal work protocol, internet disabled, GPT5.5 medium and Sonnet4.6 judges. Task pass requires every criterion; final is mean of two judges. Criteria-pass rate is separate. DOCX tracked-change grading fix. Distinct from AA hallucination-gated LAB. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":2901.7,"stderr":1.286,"cost_per_test":6.272163,"temperature":1.0,"top_p":null,"max_output_tokens":131072,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"Meta","harness":null}

96.078% · raw 96.078 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Harvey held-out legal work protocol, internet disabled, GPT5.5 medium and Sonnet4.6 judges. Task pass requires every criterion; final is mean of two judges. Criteria-pass rate is separate. DOCX tracked-change grading fix. Distinct from AA hallucination-gated LAB. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":2353.468,"stderr":1.218,"cost_per_test":4.235198,"temperature":1.0,"top_p":null,"max_output_tokens":131072,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"Meta","harness":null}

94.692% · raw 94.692 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Harvey held-out legal work protocol, internet disabled, GPT5.5 medium and Sonnet4.6 judges. Task pass requires every criterion; final is mean of two judges. Criteria-pass rate is separate. DOCX tracked-change grading fix. Distinct from AA hallucination-gated LAB. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":2735.374,"stderr":1.314,"cost_per_test":5.218937,"temperature":1.0,"top_p":null,"max_output_tokens":131072,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"Meta","harness":null}

97.588% · raw 97.588 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Harvey held-out legal work protocol, internet disabled, GPT5.5 medium and Sonnet4.6 judges. Task pass requires every criterion; final is mean of two judges. Criteria-pass rate is separate. DOCX tracked-change grading fix. Distinct from AA hallucination-gated LAB. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":2894.524,"stderr":0.871,"cost_per_test":5.165325,"temperature":1.0,"top_p":null,"max_output_tokens":131072,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"Meta","harness":null}

98.057% · raw 98.057 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Harvey held-out legal work protocol, internet disabled, GPT5.5 medium and Sonnet4.6 judges. Task pass requires every criterion; final is mean of two judges. Criteria-pass rate is separate. DOCX tracked-change grading fix. Distinct from AA hallucination-gated LAB. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":2553.994,"stderr":0.822,"cost_per_test":5.724329,"temperature":1.0,"top_p":null,"max_output_tokens":131072,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"Meta","harness":null}

97.099% · raw 97.099 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Harvey held-out legal work protocol, internet disabled, GPT5.5 medium and Sonnet4.6 judges. Task pass requires every criterion; final is mean of two judges. Criteria-pass rate is separate. DOCX tracked-change grading fix. Distinct from AA hallucination-gated LAB. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":2502.37,"stderr":0.982,"cost_per_test":6.394651,"temperature":1.0,"top_p":null,"max_output_tokens":131072,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"Meta","harness":null}

93.446% · raw 93.446 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Harvey held-out legal work protocol, internet disabled, GPT5.5 medium and Sonnet4.6 judges. Task pass requires every criterion; final is mean of two judges. Criteria-pass rate is separate. DOCX tracked-change grading fix. Distinct from AA hallucination-gated LAB. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":2510.782,"stderr":1.517,"cost_per_test":4.65707,"temperature":1.0,"top_p":null,"max_output_tokens":131072,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"Meta","harness":null}

98.185% · raw 98.185 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Harvey held-out legal work protocol, internet disabled, GPT5.5 medium and Sonnet4.6 judges. Task pass requires every criterion; final is mean of two judges. Criteria-pass rate is separate. DOCX tracked-change grading fix. Distinct from AA hallucination-gated LAB. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":3120.71,"stderr":0.768,"cost_per_test":6.264605,"temperature":1.0,"top_p":null,"max_output_tokens":131072,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"Meta","harness":null}

93.190% · raw 93.19 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Harvey held-out legal work protocol, internet disabled, GPT5.5 medium and Sonnet4.6 judges. Task pass requires every criterion; final is mean of two judges. Criteria-pass rate is separate. DOCX tracked-change grading fix. Distinct from AA hallucination-gated LAB. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":2908.808,"stderr":1.511,"cost_per_test":4.77579,"temperature":1.0,"top_p":null,"max_output_tokens":131072,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"Meta","harness":null}

93.878% · raw 93.878 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Harvey held-out legal work protocol, internet disabled, GPT5.5 medium and Sonnet4.6 judges. Task pass requires every criterion; final is mean of two judges. Criteria-pass rate is separate. DOCX tracked-change grading fix. Distinct from AA hallucination-gated LAB. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":3311.582,"stderr":1.296,"cost_per_test":6.091472,"temperature":1.0,"top_p":null,"max_output_tokens":131072,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"Meta","harness":null}

96.644% · raw 96.644 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Harvey held-out legal work protocol, internet disabled, GPT5.5 medium and Sonnet4.6 judges. Task pass requires every criterion; final is mean of two judges. Criteria-pass rate is separate. DOCX tracked-change grading fix. Distinct from AA hallucination-gated LAB. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":2539.434,"stderr":1.045,"cost_per_test":4.848495,"temperature":1.0,"top_p":null,"max_output_tokens":131072,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"Meta","harness":null}

95.455% · raw 95.455 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Harvey held-out legal work protocol, internet disabled, GPT5.5 medium and Sonnet4.6 judges. Task pass requires every criterion; final is mean of two judges. Criteria-pass rate is separate. DOCX tracked-change grading fix. Distinct from AA hallucination-gated LAB. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":2604.304,"stderr":1.258,"cost_per_test":4.272067,"temperature":1.0,"top_p":null,"max_output_tokens":131072,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"Meta","harness":null}

95.956% · raw 95.956 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Harvey held-out legal work protocol, internet disabled, GPT5.5 medium and Sonnet4.6 judges. Task pass requires every criterion; final is mean of two judges. Criteria-pass rate is separate. DOCX tracked-change grading fix. Distinct from AA hallucination-gated LAB. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":2466.874,"stderr":1.197,"cost_per_test":4.928353,"temperature":1.0,"top_p":null,"max_output_tokens":131072,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"Meta","harness":null}

97.865% · raw 97.865 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Harvey held-out legal work protocol, internet disabled, GPT5.5 medium and Sonnet4.6 judges. Task pass requires every criterion; final is mean of two judges. Criteria-pass rate is separate. DOCX tracked-change grading fix. Distinct from AA hallucination-gated LAB. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":2474.152,"stderr":0.864,"cost_per_test":5.894615,"temperature":1.0,"top_p":null,"max_output_tokens":131072,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"Meta","harness":null}

97.211% · raw 97.211 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Harvey held-out legal work protocol, internet disabled, GPT5.5 medium and Sonnet4.6 judges. Task pass requires every criterion; final is mean of two judges. Criteria-pass rate is separate. DOCX tracked-change grading fix. Distinct from AA hallucination-gated LAB. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":2063.232,"stderr":1.041,"cost_per_test":4.26355,"temperature":1.0,"top_p":null,"max_output_tokens":131072,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"Meta","harness":null}

95.113% · raw 95.113 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Harvey held-out legal work protocol, internet disabled, GPT5.5 medium and Sonnet4.6 judges. Task pass requires every criterion; final is mean of two judges. Criteria-pass rate is separate. DOCX tracked-change grading fix. Distinct from AA hallucination-gated LAB. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":2486.446,"stderr":1.324,"cost_per_test":5.721406,"temperature":1.0,"top_p":null,"max_output_tokens":131072,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"Meta","harness":null}

98.397% · raw 98.397 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Harvey held-out legal work protocol, internet disabled, GPT5.5 medium and Sonnet4.6 judges. Task pass requires every criterion; final is mean of two judges. Criteria-pass rate is separate. DOCX tracked-change grading fix. Distinct from AA hallucination-gated LAB. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":2871.286,"stderr":0.679,"cost_per_test":6.578616,"temperature":1.0,"top_p":null,"max_output_tokens":131072,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"Meta","harness":null}

96.129% · raw 96.129 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Harvey held-out legal work protocol, internet disabled, GPT5.5 medium and Sonnet4.6 judges. Task pass requires every criterion; final is mean of two judges. Criteria-pass rate is separate. DOCX tracked-change grading fix. Distinct from AA hallucination-gated LAB. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":2426.824,"stderr":1.097,"cost_per_test":6.8176,"temperature":1.0,"top_p":null,"max_output_tokens":131072,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"Meta","harness":null}

92.642% · raw 92.642 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Harvey held-out legal work protocol, internet disabled, GPT5.5 medium and Sonnet4.6 judges. Task pass requires every criterion; final is mean of two judges. Criteria-pass rate is separate. DOCX tracked-change grading fix. Distinct from AA hallucination-gated LAB. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":2475.476,"stderr":1.512,"cost_per_test":5.723511,"temperature":1.0,"top_p":null,"max_output_tokens":131072,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"Meta","harness":null}

96.565% · raw 96.565 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Harvey held-out legal work protocol, internet disabled, GPT5.5 medium and Sonnet4.6 judges. Task pass requires every criterion; final is mean of two judges. Criteria-pass rate is separate. DOCX tracked-change grading fix. Distinct from AA hallucination-gated LAB. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":2998.392,"stderr":1.127,"cost_per_test":4.090553,"temperature":1.0,"top_p":null,"max_output_tokens":131072,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"Meta","harness":null}

96.633% · raw 96.633 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Harvey held-out legal work protocol, internet disabled, GPT5.5 medium and Sonnet4.6 judges. Task pass requires every criterion; final is mean of two judges. Criteria-pass rate is separate. DOCX tracked-change grading fix. Distinct from AA hallucination-gated LAB. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":2595.43,"stderr":1.048,"cost_per_test":5.453971,"temperature":1.0,"top_p":null,"max_output_tokens":131072,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"Meta","harness":null}

$2.147235/task · raw 2.147235 USD/task

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Private193-item tax suite. All-pass accuracy and must-pass weighted partial credit are separate; both multiply by citation factor0.7+0.3*citation quality. No inferred tax professional reliability. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":1084.296,"stderr":3.572,"cost_per_test":2.147235,"token_totals":{"input_tokens":257606995,"output_tokens":9968530,"reasoning_tokens":6086406,"cache_read_tokens":214574250,"cache_write_tokens":0},"temperature":1.0,"top_p":null,"max_output_tokens":131072,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"Meta","harness":null} Source score standard error describes the benchmark quality score, not uncertainty in cost or latency.

1084.296 s · raw 1084.296 s

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Private193-item tax suite. All-pass accuracy and must-pass weighted partial credit are separate; both multiply by citation factor0.7+0.3*citation quality. No inferred tax professional reliability. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":1084.296,"stderr":3.572,"cost_per_test":2.147235,"token_totals":{"input_tokens":257606995,"output_tokens":9968530,"reasoning_tokens":6086406,"cache_read_tokens":214574250,"cache_write_tokens":0},"temperature":1.0,"top_p":null,"max_output_tokens":131072,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"Meta","harness":null} Source score standard error describes the benchmark quality score, not uncertainty in cost or latency.

46.512% · raw 46.512 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Private193-item tax suite. All-pass accuracy and must-pass weighted partial credit are separate; both multiply by citation factor0.7+0.3*citation quality. No inferred tax professional reliability. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":1141.924,"stderr":7.696,"cost_per_test":2.157843,"temperature":1.0,"top_p":null,"max_output_tokens":131072,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"Meta","harness":null}

50.000% · raw 50 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Private193-item tax suite. All-pass accuracy and must-pass weighted partial credit are separate; both multiply by citation factor0.7+0.3*citation quality. No inferred tax professional reliability. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":972.116,"stderr":9.285,"cost_per_test":1.973979,"temperature":1.0,"top_p":null,"max_output_tokens":131072,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"Meta","harness":null}

56.614% · raw 56.614 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Private193-item tax suite. All-pass accuracy and must-pass weighted partial credit are separate; both multiply by citation factor0.7+0.3*citation quality. No inferred tax professional reliability. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":1369.127,"stderr":9.193,"cost_per_test":2.742275,"temperature":1.0,"top_p":null,"max_output_tokens":131072,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"Meta","harness":null}

46.667% · raw 46.667 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Private193-item tax suite. All-pass accuracy and must-pass weighted partial credit are separate; both multiply by citation factor0.7+0.3*citation quality. No inferred tax professional reliability. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":455.508,"stderr":9.264,"cost_per_test":0.876776,"temperature":1.0,"top_p":null,"max_output_tokens":131072,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"Meta","harness":null}

30.000% · raw 30 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Private193-item tax suite. All-pass accuracy and must-pass weighted partial credit are separate; both multiply by citation factor0.7+0.3*citation quality. No inferred tax professional reliability. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":1264.709,"stderr":8.51,"cost_per_test":2.488422,"temperature":1.0,"top_p":null,"max_output_tokens":131072,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"Meta","harness":null}

26.667% · raw 26.667 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Private193-item tax suite. All-pass accuracy and must-pass weighted partial credit are separate; both multiply by citation factor0.7+0.3*citation quality. No inferred tax professional reliability. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":1309.206,"stderr":8.212,"cost_per_test":2.63952,"temperature":1.0,"top_p":null,"max_output_tokens":131072,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"Meta","harness":null}

74.764% · raw 74.764 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Private193-item tax suite. All-pass accuracy and must-pass weighted partial credit are separate; both multiply by citation factor0.7+0.3*citation quality. No inferred tax professional reliability. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":1141.924,"stderr":6.372,"cost_per_test":2.157843,"temperature":1.0,"top_p":null,"max_output_tokens":131072,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"Meta","harness":null}

68.105% · raw 68.105 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Private193-item tax suite. All-pass accuracy and must-pass weighted partial credit are separate; both multiply by citation factor0.7+0.3*citation quality. No inferred tax professional reliability. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":972.116,"stderr":8.313,"cost_per_test":1.973979,"temperature":1.0,"top_p":null,"max_output_tokens":131072,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"Meta","harness":null}

78.312% · raw 78.312 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Private193-item tax suite. All-pass accuracy and must-pass weighted partial credit are separate; both multiply by citation factor0.7+0.3*citation quality. No inferred tax professional reliability. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":1369.127,"stderr":7.299,"cost_per_test":2.742275,"temperature":1.0,"top_p":null,"max_output_tokens":131072,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"Meta","harness":null}

76.278% · raw 76.278 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Private193-item tax suite. All-pass accuracy and must-pass weighted partial credit are separate; both multiply by citation factor0.7+0.3*citation quality. No inferred tax professional reliability. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":455.508,"stderr":7.154,"cost_per_test":0.876776,"temperature":1.0,"top_p":null,"max_output_tokens":131072,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"Meta","harness":null}

76.552% · raw 76.552 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Private193-item tax suite. All-pass accuracy and must-pass weighted partial credit are separate; both multiply by citation factor0.7+0.3*citation quality. No inferred tax professional reliability. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":1264.709,"stderr":7.157,"cost_per_test":2.488422,"temperature":1.0,"top_p":null,"max_output_tokens":131072,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"Meta","harness":null}

76.097% · raw 76.097 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Private193-item tax suite. All-pass accuracy and must-pass weighted partial credit are separate; both multiply by citation factor0.7+0.3*citation quality. No inferred tax professional reliability. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":1309.206,"stderr":7.11,"cost_per_test":2.63952,"temperature":1.0,"top_p":null,"max_output_tokens":131072,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"Meta","harness":null}

85.996% · raw 85.996 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: OpenHands. Independent model-plus-harness web-application evaluation. Preserve OpenHands versus other agent variants; source costs/runtime are task observations, not API tariffs. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":648.336,"stderr":1.933,"cost_per_test":2.293448,"token_totals":{"input_tokens":222902077,"output_tokens":2071146,"reasoning_tokens":144351,"cache_read_tokens":215357588,"cache_write_tokens":0},"temperature":1.0,"top_p":null,"max_output_tokens":131072,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"Meta","harness":"OpenHands"}

86.000% · raw 86 %

Alternative · max effort · Independent evaluator

Source/record date: 2026-10-12

Vals exact Meta Wonder identity, model page linked by 12 October v2 Finance/Index boards; max effort. Default Meta provider, temperature 1, top_p/top_k default, 131,072 output-token harness ceiling; suite overrides possible. Run date unpublished. Published uncertainty ±1.93 percentage points. Citation repaired 2026-10-14: original https://www.vals.ai/models/meta_straw_110_max now returns 404; the exact renamed model page https://www.vals.ai/models/meta_muse-wonder-full reproduces this recorded value at its published precision. Historical score, effort and as_of are retained. Demoted 2026-10-20: current publisher observation from https://www.vals.ai/benchmarks/vibe-code is the headline for this same version/configuration family; this historical value and source remain available. Revised benchmark versions use separate metrics.

$2.293448/task · raw 2.293448 USD/task

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: OpenHands. Independent model-plus-harness web-application evaluation. Preserve OpenHands versus other agent variants; source costs/runtime are task observations, not API tariffs. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":648.336,"stderr":1.933,"cost_per_test":2.293448,"token_totals":{"input_tokens":222902077,"output_tokens":2071146,"reasoning_tokens":144351,"cache_read_tokens":215357588,"cache_write_tokens":0},"temperature":1.0,"top_p":null,"max_output_tokens":131072,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"Meta","harness":"OpenHands"} Source score standard error describes the benchmark quality score, not uncertainty in cost or latency.

$2.29/task · raw 2.29 USD/task

Alternative · max effort · Independent evaluator

Source/record date: 2026-10-12

Published evaluator Cost / Test; max default, suite-specific harness. Vibe Code uses OpenHands. Not token prices or subscription charges; Full priced evaluator run does not establish a public API. Demoted 2026-10-20: current publisher observation from https://www.vals.ai/benchmarks/vibe-code is the headline for this same version/configuration family; this historical value and source remain available. Revised benchmark versions use separate metrics.

648.336 s · raw 648.336 s

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: OpenHands. Independent model-plus-harness web-application evaluation. Preserve OpenHands versus other agent variants; source costs/runtime are task observations, not API tariffs. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":648.336,"stderr":1.933,"cost_per_test":2.293448,"token_totals":{"input_tokens":222902077,"output_tokens":2071146,"reasoning_tokens":144351,"cache_read_tokens":215357588,"cache_write_tokens":0},"temperature":1.0,"top_p":null,"max_output_tokens":131072,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"Meta","harness":"OpenHands"} Source score standard error describes the benchmark quality score, not uncertainty in cost or latency.

1857 · raw 1857 Elo

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Manually observed public GDPval-AA v2.1 leaderboard table, full 298-configuration pool checked 20 October 2026. Exact source configuration: Muse Wonder Full (Max); creator: Meta; source rank 2; source release-month label Oct 2026 is model release metadata, not execution date. Published rounded native Elo 1857; literal displayed CI -25 / +25. Native Elo is not a percentage and is not the normalized (Elo−500)/2000 chart quantity. This versioned metric is separate from older generic GDPval-AA Elo and vendor GDPval/GDPval-AA results; no old evidence is overwritten. Execution date, inference precision, unprinted effort budgets and task costs are unpublished in this captured table. Elo is relative to the dated comparison pool; drift from an earlier observation does not prove a new evaluation run. Headline: highest published native Elo for this exact approved model; tied values prefer existing featured effort then source order.

55.8 · raw 55.802119985562 index

Headline · max effort · Independent evaluator

Source/record date: 2026-10-12

AA exact public model-page data, max effort; checked 12 October 2026, run date unpublished. Intelligence Index v4.3.2; non-estimated. Native index/Elo scale.

1857.1 · raw 1857.09 Elo

Headline · max effort · Independent evaluator

Source/record date: 2026-10-12

AA exact public model-page data, max effort; checked 12 October 2026, run date unpublished. Intelligence Index v4.3.2; non-estimated. Native index/Elo scale.

30.2 · raw 30.15 index

Headline · max effort · Independent evaluator

Source/record date: 2026-10-12

AA exact public model-page data, max effort; checked 12 October 2026, run date unpublished. Intelligence Index v4.3.2; non-estimated. Native index/Elo scale.

43.2% · raw 43.216666666667 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-12

AA exact public model-page data, max effort; checked 12 October 2026, run date unpublished. Intelligence Index v4.3.2; non-estimated. Raw 0–1 score ×100.

23% · raw 23.011447020839 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-12

AA exact public model-page data, max effort; checked 12 October 2026, run date unpublished. Intelligence Index v4.3.2; non-estimated. Raw 0–1 score ×100.

57.1% · raw 57.070707070707 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-12

AA exact public model-page data, max effort; checked 12 October 2026, run date unpublished. Intelligence Index v4.3.2; non-estimated. Raw 0–1 score ×100.

62.6% · raw 62.615740740741 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-12

AA exact public model-page data, max effort; checked 12 October 2026, run date unpublished. Intelligence Index v4.3.2; non-estimated. Raw 0–1 score ×100. Evaluation marked under review on source.

83.7% · raw 83.666666666667 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-12

AA exact public model-page data, max effort; checked 12 October 2026, run date unpublished. Intelligence Index v4.3.2; non-estimated. Raw 0–1 score ×100.

53.6% · raw 53.614457831325 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-12

AA exact public model-page data, max effort; checked 12 October 2026, run date unpublished. Intelligence Index v4.3.2; non-estimated. Raw 0–1 score ×100.

32.6% · raw 32.6 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-12

AA exact public model-page data, max effort; checked 12 October 2026, run date unpublished. Intelligence Index v4.3.2; non-estimated. Raw 0–1 score ×100.

32.9% · raw 32.857142857143 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-12

AA exact public model-page data, max effort; checked 12 October 2026, run date unpublished. Intelligence Index v4.3.2; non-estimated. Raw 0–1 score ×100. Evaluation marked under review on source.

71.3% · raw 71.33074973224 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-12

AA exact public model-page data, max effort; checked 12 October 2026, run date unpublished. Intelligence Index v4.3.2; non-estimated. Raw 0–1 score ×100. Partial reward, not strict all-assertion completion (Meta quotes 39.1/45.4 strict).

54.2 · raw 54.152003084622 index

Headline · max effort · Independent evaluator

Source/record date: 2026-10-12

AA exact public model-page data, max effort; checked 12 October 2026, run date unpublished. Intelligence Index v4.3.2; non-estimated. Native capability index; not percent.

57.8 · raw 57.767241011047 index

Headline · max effort · Independent evaluator

Source/record date: 2026-10-12

AA exact public model-page data, max effort; checked 12 October 2026, run date unpublished. Intelligence Index v4.3.2; non-estimated. Native capability index; not percent.

54.7 · raw 54.73463646402 index

Headline · max effort · Independent evaluator

Source/record date: 2026-10-12

AA exact public model-page data, max effort; checked 12 October 2026, run date unpublished. Intelligence Index v4.3.2; non-estimated. Native capability index; not percent.

57 · raw 56.968396163876 index

Headline · max effort · Independent evaluator

Source/record date: 2026-10-12

AA exact public model-page data, max effort; checked 12 October 2026, run date unpublished. Intelligence Index v4.3.2; non-estimated. Native capability index; not percent.

60.7 · raw 60.725956074297 index

Headline · max effort · Independent evaluator

Source/record date: 2026-10-12

AA exact public model-page data, max effort; checked 12 October 2026, run date unpublished. Intelligence Index v4.3.2; non-estimated. Native capability index; not percent.

1729.3 · raw 1729.32 Elo

Headline · max effort · Independent evaluator

Source/record date: 2026-10-12

AA exact public model-page data, max effort; checked 12 October 2026, run date unpublished. Intelligence Index v4.3.2; non-estimated. AA-Briefcase v1.1; rubric fraction converted to percent, quality/presentation retain Elo.

61.3% · raw 61.313131313131 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-12

AA exact public model-page data, max effort; checked 12 October 2026, run date unpublished. Intelligence Index v4.3.2; non-estimated. AA-Briefcase v1.1; rubric fraction converted to percent, quality/presentation retain Elo.

2116.3 · raw 2116.26 Elo

Headline · max effort · Independent evaluator

Source/record date: 2026-10-12

AA exact public model-page data, max effort; checked 12 October 2026, run date unpublished. Intelligence Index v4.3.2; non-estimated. AA-Briefcase v1.1; rubric fraction converted to percent, quality/presentation retain Elo.

1524.1 · raw 1524.05 Elo

Headline · max effort · Independent evaluator

Source/record date: 2026-10-12

AA exact public model-page data, max effort; checked 12 October 2026, run date unpublished. Intelligence Index v4.3.2; non-estimated. AA-Briefcase v1.1; rubric fraction converted to percent, quality/presentation retain Elo.

64.570% · raw 64.57 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-12

Vals exact Meta Wonder identity, model page linked by 12 October v2 Finance/Index boards; max effort. Default Meta provider, temperature 1, top_p/top_k default, 131,072 output-token harness ceiling; suite overrides possible. Run date unpublished. Published uncertainty ±1.07 percentage points. Citation repaired 2026-10-14: original https://www.vals.ai/models/meta_straw_110_max now returns 404; the exact renamed model page https://www.vals.ai/models/meta_muse-wonder-full reproduces this recorded value at its published precision. Historical score, effort and as_of are retained.

61% · raw 61.03 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-12

Vals exact Meta Wonder identity, model page linked by 12 October v2 Finance/Index boards; max effort. Default Meta provider, temperature 1, top_p/top_k default, 131,072 output-token harness ceiling; suite overrides possible. Run date unpublished. Published uncertainty ±4.42 percentage points. Citation repaired 2026-10-14: original https://www.vals.ai/models/meta_straw_110_max now returns 404; the exact renamed model page https://www.vals.ai/models/meta_muse-wonder-full reproduces this recorded value at its published precision. Historical score, effort and as_of are retained.

73.6% · raw 73.63 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-12

Vals exact Meta Wonder identity, model page linked by 12 October v2 Finance/Index boards; max effort. Default Meta provider, temperature 1, top_p/top_k default, 131,072 output-token harness ceiling; suite overrides possible. Run date unpublished. Published uncertainty ±5.56 percentage points. Vals CyberBench v1.1 capability, not Meta refusal CyberBench. Citation repaired 2026-10-14: original https://www.vals.ai/models/meta_straw_110_max now returns 404; the exact renamed model page https://www.vals.ai/models/meta_muse-wonder-full reproduces this recorded value at its published precision. Historical score, effort and as_of are retained.

71.9% · raw 71.92 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-12

Vals exact Meta Wonder identity, model page linked by 12 October v2 Finance/Index boards; max effort. Default Meta provider, temperature 1, top_p/top_k default, 131,072 output-token harness ceiling; suite overrides possible. Run date unpublished. Published uncertainty ±2.66 percentage points. Citation repaired 2026-10-14: original https://www.vals.ai/models/meta_straw_110_max now returns 404; the exact renamed model page https://www.vals.ai/models/meta_muse-wonder-full reproduces this recorded value at its published precision. Historical score, effort and as_of are retained.

59.1% · raw 59.06 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-12

Vals exact Meta Wonder identity, model page linked by 12 October v2 Finance/Index boards; max effort. Default Meta provider, temperature 1, top_p/top_k default, 131,072 output-token harness ceiling; suite overrides possible. Run date unpublished. Published uncertainty ±0.21 percentage points. Citation repaired 2026-10-14: original https://www.vals.ai/models/meta_straw_110_max now returns 404; the exact renamed model page https://www.vals.ai/models/meta_muse-wonder-full reproduces this recorded value at its published precision. Historical score, effort and as_of are retained.

53.4% · raw 53.37 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-12

Vals exact Meta Wonder identity, model page linked by 12 October v2 Finance/Index boards; max effort. Default Meta provider, temperature 1, top_p/top_k default, 131,072 output-token harness ceiling; suite overrides possible. Run date unpublished. Published uncertainty ±3.47 percentage points. Citation repaired 2026-10-14: original https://www.vals.ai/models/meta_straw_110_max now returns 404; the exact renamed model page https://www.vals.ai/models/meta_muse-wonder-full reproduces this recorded value at its published precision. Historical score, effort and as_of are retained.

35.1% · raw 35.13 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-12

Vals exact Meta Wonder identity, model page linked by 12 October v2 Finance/Index boards; max effort. Default Meta provider, temperature 1, top_p/top_k default, 131,072 output-token harness ceiling; suite overrides possible. Run date unpublished. Published uncertainty ±3.21 percentage points. Citation repaired 2026-10-14: original https://www.vals.ai/models/meta_straw_110_max now returns 404; the exact renamed model page https://www.vals.ai/models/meta_muse-wonder-full reproduces this recorded value at its published precision. Historical score, effort and as_of are retained.

94% · raw 94 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-12

Vals exact Meta Wonder identity, model page linked by 12 October v2 Finance/Index boards; max effort. Default Meta provider, temperature 1, top_p/top_k default, 131,072 output-token harness ceiling; suite overrides possible. Run date unpublished. Published uncertainty ±2.39 percentage points. Citation repaired 2026-10-14: original https://www.vals.ai/models/meta_straw_110_max now returns 404; the exact renamed model page https://www.vals.ai/models/meta_muse-wonder-full reproduces this recorded value at its published precision. Historical score, effort and as_of are retained.

73.8% · raw 73.75 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-12

Vals exact Meta Wonder identity, model page linked by 12 October v2 Finance/Index boards; max effort. Default Meta provider, temperature 1, top_p/top_k default, 131,072 output-token harness ceiling; suite overrides possible. Run date unpublished. Published uncertainty ±1.15 percentage points. Citation repaired 2026-10-14: original https://www.vals.ai/models/meta_straw_110_max now returns 404; the exact renamed model page https://www.vals.ai/models/meta_muse-wonder-full reproduces this recorded value at its published precision. Historical score, effort and as_of are retained.

43% · raw 43 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-12

Vals exact Meta Wonder identity, model page linked by 12 October v2 Finance/Index boards; max effort. Default Meta provider, temperature 1, top_p/top_k default, 131,072 output-token harness ceiling; suite overrides possible. Run date unpublished. Published uncertainty ±3.57 percentage points. Reviewed 2026-10-14: this legacy generic key mixes quantity labels across models. Current source explicitly identifies this historical value as Accuracy (published current raw 42.997). Keep stored metric/value unchanged for historical evidence; use newly separate citation-adjusted Accuracy and Weighted Pass metrics for like-for-like comparisons. No inferred model regression. Citation repaired 2026-10-14: original https://www.vals.ai/models/meta_straw_110_max now returns 404; the exact renamed model page https://www.vals.ai/models/meta_muse-wonder-full reproduces this recorded value at its published precision. Historical score, effort and as_of are retained.

73.3% · raw 73.33 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-12

Vals exact Meta Wonder identity, model page linked by 12 October v2 Finance/Index boards; max effort. Default Meta provider, temperature 1, top_p/top_k default, 131,072 output-token harness ceiling; suite overrides possible. Run date unpublished. Published uncertainty ±0.64 percentage points. Citation repaired 2026-10-14: original https://www.vals.ai/models/meta_straw_110_max now returns 404; the exact renamed model page https://www.vals.ai/models/meta_muse-wonder-full reproduces this recorded value at its published precision. Historical score, effort and as_of are retained.

21.7% · raw 21.67 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-12

Vals exact Meta Wonder identity, model page linked by 12 October v2 Finance/Index boards; max effort. Default Meta provider, temperature 1, top_p/top_k default, 131,072 output-token harness ceiling; suite overrides possible. Run date unpublished. Published uncertainty ±3.54 percentage points. Citation repaired 2026-10-14: original https://www.vals.ai/models/meta_straw_110_max now returns 404; the exact renamed model page https://www.vals.ai/models/meta_muse-wonder-full reproduces this recorded value at its published precision. Historical score, effort and as_of are retained.

79.8% · raw 79.83 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-12

Vals exact Meta Wonder identity, model page linked by 12 October v2 Finance/Index boards; max effort. Default Meta provider, temperature 1, top_p/top_k default, 131,072 output-token harness ceiling; suite overrides possible. Run date unpublished. Published uncertainty ±2.99 percentage points. Citation repaired 2026-10-14: original https://www.vals.ai/models/meta_straw_110_max now returns 404; the exact renamed model page https://www.vals.ai/models/meta_muse-wonder-full reproduces this recorded value at its published precision. Historical score, effort and as_of are retained.

0.5% · raw 0.5 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-12

Vals exact Meta Wonder identity, model page linked by 12 October v2 Finance/Index boards; max effort. Default Meta provider, temperature 1, top_p/top_k default, 131,072 output-token harness ceiling; suite overrides possible. Run date unpublished. Published uncertainty ±0.50 percentage points. Vals ProgramBench fully-resolved score; not Meta mini-swe-agent ≥95%-tests Almost score. Citation repaired 2026-10-14: original https://www.vals.ai/models/meta_straw_110_max now returns 404; the exact renamed model page https://www.vals.ai/models/meta_muse-wonder-full reproduces this recorded value at its published precision. Historical score, effort and as_of are retained.

58.1% · raw 58.08 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-12

Vals exact Meta Wonder identity, model page linked by 12 October v2 Finance/Index boards; max effort. Default Meta provider, temperature 1, top_p/top_k default, 131,072 output-token harness ceiling; suite overrides possible. Run date unpublished. Published uncertainty ±3.31 percentage points. Citation repaired 2026-10-14: original https://www.vals.ai/models/meta_straw_110_max now returns 404; the exact renamed model page https://www.vals.ai/models/meta_muse-wonder-full reproduces this recorded value at its published precision. Historical score, effort and as_of are retained.

$8.559/task · raw 8.559 USD/task

Headline · max effort · Independent evaluator

Source/record date: 2026-10-12

Vals observed index cost/test; max effort. Model page full precision; dated evaluator valuation, not public API availability or a subscription bill. Run date unpublished. Citation repaired 2026-10-14: original https://www.vals.ai/models/meta_straw_110_max now returns 404; the exact renamed model page https://www.vals.ai/models/meta_muse-wonder-full reproduces this recorded value at its published precision. Historical score, effort and as_of are retained.

$2.39/task · raw 2.39 USD/task

Headline · max effort · Independent evaluator

Source/record date: 2026-10-12

Published evaluator Cost / Test; max default, suite-specific harness. Vibe Code uses OpenHands. Not token prices or subscription charges; Full priced evaluator run does not establish a public API.

$11791.56 · raw 11791.56 USD

Headline · unknown effort · Independent evaluator

Source/record date: 2026-10-14

Harness: Andon Vending-Bench 2; exact public configuration Muse Wonder Full. Source runs.vb 2.final_value is the publisher's arithmetic mean final bank balance over 6 year-long simulated runs, not a new average created by Examenos. Net money gain = published mean final balance $12291.558333333334 minus the benchmark's $500 starting balance; $11791.56 net gain, rounded to cents. Mean final-balance std 2167.231184155643; SEM 884.7684259715148; 95% CI half-width 1734.146114904169. Preserve standard deviation and CI semantics. This is simulated business money, not API cost, real revenue or subscription value. Source model effort and execution dates are not published in this data module. Vending-Bench 1 data are excluded.

42 fixes · raw 42 fixes

Headline · max effort · Independent evaluator

Source/record date: 2026-10-16

Harness: Muse Code / Meta API. Current source two-run mean; Meta's Muse Wonder Full 1.0 at max effort through Muse Code, Meta's own agent CLI. MEAN OF TWO RUNS: 44 and 40 of 105 (repo 1: 22 and 21 of 45 - repo 2: 22 and 19 of 60), mean 42.0. Genuine unplanted extras 53 and 20; claimed-only fixes 0 and 0. Active minutes 138.2 and 152.3. *** XHIGH, ALSO A MEAN OF TWO *** stands at 41.5 (16.0 on repo 1, 25.5 on repo 2). One run each had tied the tiers at 44; two runs each put max narrowly ahead. The tiers split the repos consistently: max leads on repo 1, xhigh on repo 2. *** RUN 2 RAN IN SEGMENTS *** It was interrupted - by the subscription's usage windows, by an operator stop and by a closed console - and each time Muse Code continued the same session. Its wall counts active time only: each segment from its first model call to its last before any refusal; every pause is excluded. Run 2 also straddles a harness update: Muse Code 1.4.1 for its first segment(s), 1.4.7 after the launcher updated itself. *** COST UNKNOWN *** Meta publishes no rate for Full; the cost column is blank, not estimated. *** EFFORT max *** the top of the effort list Muse Code's catalog gives this model; not magnitude-probed. The tier describes Muse Code's main loop. *** RECEIPTS *** 2 of the 4 legs shared the laptop with another arm and the rest had it to themselves, which the wall marker discloses per leg - scores, tokens and cost are untouched by that, only the clock. Judge: blind Codex gpt-5.5, which is neither this model nor a sibling of it. Mean of two runs (44 and 40 of 105). Run 2 ran in resumed segments; its time is active time only, and it spans Muse Code 1.4.1 and 1.4.7. Cost unknown: Meta publishes no price for this model. Model run dates are not inferred from this aggregate row date. Only planted fixes count; no cost is invented. Checked 16 October 2026.

44 fixes · raw 44 fixes

Alternative · max effort · Independent evaluator

Source/record date: 2026-10-14

Harness: Muse Code 1.4.1 / Meta API; exact Muse Wonder Full 1.0 max configuration, n 1, source row 13 Oct 2026, checked 14 Oct. Planted fixes 44/105; native effort status first_party. This single run is not a replication mean. Max and xhigh are separate source configurations despite equal observed scores. Only planted fixes count. Keep the source's requested/verified effort qualification; no conversion to a provider allowance, bill or general workload cost. Superseded 16 October 2026 by current two-run max mean 42 from the same board; original single-run value and observation date preserved.

41.5 fixes · raw 41.5 fixes

Alternative · xhigh effort · Independent evaluator

Source/record date: 2026-10-16

Harness: Muse Code / Meta API. Current source two-run mean; Meta's Muse Wonder Full 1.0 at xhigh effort through Muse Code, Meta's own agent CLI. MEAN OF TWO RUNS: 44 and 39 of 105 (repo 1: 16 and 16 of 45 - repo 2: 28 and 23 of 60), mean 41.5. Genuine unplanted extras 35 and 14; claimed-only fixes 0 and 0. Active minutes 80.3 and 100.8. *** MAX, ALSO A MEAN OF TWO *** stands at 42.0 (21.5 on repo 1, 20.5 on repo 2). One run each had tied the tiers at 44; two runs each put max narrowly ahead. The tiers split the repos consistently: max leads on repo 1, xhigh on repo 2. *** RUN 2 RAN IN SEGMENTS *** It was interrupted - by the subscription's usage windows, by an operator stop and by a closed console - and each time Muse Code continued the same session. Its wall counts active time only: each segment from its first model call to its last before any refusal; every pause is excluded. Run 2 also straddles a harness update: Muse Code 1.4.1 for its first segment(s), 1.4.7 after the launcher updated itself. *** COST UNKNOWN *** Meta publishes no rate for Full; the cost column is blank, not estimated. *** EFFORT xhigh *** one tier below the top of the effort list Muse Code's catalog gives this model; not magnitude-probed. The tier describes Muse Code's main loop. *** RECEIPTS *** 2 of the 4 legs shared the laptop with another arm and the rest had it to themselves, which the wall marker discloses per leg - scores, tokens and cost are untouched by that, only the clock. Judge: blind Codex gpt-5.5, which is neither this model nor a sibling of it. Mean of two runs (44 and 39 of 105). Run 2 ran in resumed segments; its time is active time only, and it spans Muse Code 1.4.1 and 1.4.7. Cost unknown: Meta publishes no price for this model. Model run dates are not inferred from this aggregate row date. Only planted fixes count; no cost is invented. Checked 16 October 2026.

44 fixes · raw 44 fixes

Alternative · xhigh effort · Independent evaluator

Source/record date: 2026-10-14

Harness: Muse Code 1.4.1 / Meta API; exact Muse Wonder Full 1.0 xhigh configuration, n 1, source row 13 Oct 2026, checked 14 Oct. Planted fixes 44/105; native effort status first_party. This single run is not a replication mean. Max and xhigh are separate source configurations despite equal observed scores. Only planted fixes count. Keep the source's requested/verified effort qualification; no conversion to a provider allowance, bill or general workload cost. Superseded 16 October 2026 by current two-run xhigh mean 41.5 from the same board; original single-run value and observation date preserved.

37.1% · raw 37.143 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-14

Vals exact model page https://www.vals.ai/models/meta_muse-wonder-full; source label Terminal-Bench Science, 37.14%, published uncertainty ±5.82 percentage points. Default provider Meta, effort max, max-output harness ceiling 131,072; individual suites can override defaults. Model page defaults are not inferred vendor product limits. Suite revision/runner is checked against its primary benchmark page. Run execution date unpublished. Current suite page explicitly version 0.1; distinct from AA/Hermes harnesses. Primary suite Terminal-Bench Science v 0.1, public dataset, agentic runner external; primary updated 2026-10-13. Harness: Mini-SWE-agent.

13.1% · raw 13.1 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-14

Harness: Artificial Analysis Stirrup, Harvey private LAB v 1.1 120 tasks /24 practice areas, pass@1, up to 200 turns. Published exact configuration Muse Wonder Full (Max). Three rubric judges GPT-6 Sol/Grok 4.7/Claude Opus 5.5 medium; material hallucination checks GPT-6 Sol high. Material flags zero a task, minor flags do not. Current visible source table has 27 configurations, checked 14 Oct 2026; run date unpublished. This is independent Harvey v 1.1, not legacy HLAB(AA), Vals HLAB or Meta vendor-cited Harvey. Curated existing-metric refresh for current owner request; no future AA automated adapter authorization.

95.5% · raw 95.5 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-14

Harness: Artificial Analysis Stirrup, Harvey private LAB v 1.1 120 tasks /24 practice areas, pass@1, up to 200 turns. Published exact configuration Muse Wonder Full (Max). Three rubric judges GPT-6 Sol/Grok 4.7/Claude Opus 5.5 medium; material hallucination checks GPT-6 Sol high. Material flags zero a task, minor flags do not. Current visible source table has 27 configurations, checked 14 Oct 2026; run date unpublished. This is independent Harvey v 1.1, not legacy HLAB(AA), Vals HLAB or Meta vendor-cited Harvey. Curated existing-metric refresh for current owner request; no future AA automated adapter authorization.

0.62 hallucinations/task · raw 0.62 hallucinations/task

Headline · max effort · Independent evaluator

Source/record date: 2026-10-14

Harness: Artificial Analysis Stirrup, Harvey private LAB v 1.1 120 tasks /24 practice areas, pass@1, up to 200 turns. Published exact configuration Muse Wonder Full (Max). Three rubric judges GPT-6 Sol/Grok 4.7/Claude Opus 5.5 medium; material hallucination checks GPT-6 Sol high. Material flags zero a task, minor flags do not. Current visible source table has 27 configurations, checked 14 Oct 2026; run date unpublished. This is independent Harvey v 1.1, not legacy HLAB(AA), Vals HLAB or Meta vendor-cited Harvey. Curated existing-metric refresh for current owner request; no future AA automated adapter authorization. Mean material flags per checked usable submission, native count; not percent.

42.997% · raw 42.997 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-14

Harness: Vals Tax Agent Bench v 1, private 193-question corporate-tax test set; all questions equally weighted,3-89 rubric checks/question with 1-3 importance weights. Current source 14 Oct 2026, primary updated 12 Oct. Only final answer/citations graded; timed-out unanswered question zero. Citation factor=0.7+0.3*fraction of cited sources resolving to real authoritative documents. Accuracy requires every rubric check passing; any failed check zeroes task. Weighted Pass is weighted rubric points earned times citation factor, with any must-pass failure zeroing task (approximately 40% of checks must-pass). LLM-judged, separate from real professional correctness. This exact metric is kept separate from generic legacy Tax Agent Bench rows which mix quantity labels. Default model effortmax; individual suite overrides possible, run date unpublished. Model card displays ±3.57 percentage points; confidence level not inferred.

75.001% · raw 75.001 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-14

Harness: Vals Tax Agent Bench v 1, private 193-question corporate-tax test set; all questions equally weighted,3-89 rubric checks/question with 1-3 importance weights. Current source 14 Oct 2026, primary updated 12 Oct. Only final answer/citations graded; timed-out unanswered question zero. Citation factor=0.7+0.3*fraction of cited sources resolving to real authoritative documents. Accuracy requires every rubric check passing; any failed check zeroes task. Weighted Pass is weighted rubric points earned times citation factor, with any must-pass failure zeroing task (approximately 40% of checks must-pass). LLM-judged, separate from real professional correctness. This exact metric is kept separate from generic legacy Tax Agent Bench rows which mix quantity labels. Default model effortmax; individual suite overrides possible, run date unpublished.

48.5% · raw 48.54 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-10-17

Vals Science Index v1, published 16 October 2026. Equal weighting of BioMysteryBench, MysteryMechanism and Terminal-Bench-Science; aggregate copied directly, not computed from rounded components. Exact model label Wonder Full. Index-level effort/provider/run date not stated; do not infer from product defaults or other suites. Source auxiliary columns retained verbatim: cost/test $10.15, input/output token rates $4 / $20, input/output tokens 5.13B / 45.14M, duration 1h34m. These are evaluation context, not our public API pricing or subscription coefficients.