Examenos

42.6% · raw 42.6 %

Headline · unknown effort · Own vendor

Source/record date: 2026-10-17

Harness: DeepSeek October 17 launch comparison; evaluation date unpublished. HLE no tools, distinct from separately starred text-only results. Model-card text confirmed against the official release figure. No September-report scores transferred to the 1017 checkpoint.

44% · raw 44 %

Headline · unknown effort · Own vendor

Source/record date: 2026-10-17

Harness: DeepSeek October 17 launch comparison; evaluation date unpublished. HLE text-only subset, no tools; dagger in model card/asterisk in figure. Model-card text confirmed against the official release figure. No September-report scores transferred to the 1017 checkpoint.

42.6% · raw 42.6 %

Alternative · unknown effort · Peer vendor

Source/record date: 2026-10-19

Harness: 2,158 text-only HLE questions, no tools. As reported by MiniMax for the exact named peer; not an independent measurement. Existing headline/configuration remains preferred when present. Main figure contains no per-model effort labels. Execution date, exact checkpoint SHA, generation budget and sampling settings are unpublished. Source checked 2026-10-19.

42.3% · raw 42.3 %

Headline · unknown effort · Own vendor

Source/record date: 2026-10-17

Harness: DeepSeek October 17 launch comparison; evaluation date unpublished. Claude Code; exact version, effort and limits unspecified. Model-card text confirmed against the official release figure. No September-report scores transferred to the 1017 checkpoint.

77% · raw 77 %

Headline · max effort · Own vendor

Source/record date: 2026-10-17

Harness: DeepSeek October 17 launch comparison; evaluation date unpublished. Mini-SWE-V2, pass@1. DeepSeek max effort is explicitly labeled in the performance/cost figure; peer efforts and precise scaffold revision unspecified. Model-card text confirmed against the official release figure. No September-report scores transferred to the 1017 checkpoint.

29.1% · raw 29.1 %

Headline · unknown effort · Own vendor

Source/record date: 2026-10-17

Harness: DeepSeek October 17 launch comparison; evaluation date unpublished. ProgramBench metric/threshold/filter not restated; kept separate from Almost@1 and other published quantities. Model-card text confirmed against the official release figure. No September-report scores transferred to the 1017 checkpoint.

68.2% · raw 68.2 %

Headline · unknown effort · Own vendor

Source/record date: 2026-10-17

Harness: DeepSeek October 17 launch comparison; evaluation date unpublished. Source names this benchmark/condition but does not restate harness version, task subset or effort. Model-card text confirmed against the official release figure. No September-report scores transferred to the 1017 checkpoint.

60.5% · raw 60.5 %

Headline · unknown effort · Own vendor

Source/record date: 2026-10-17

Harness: DeepSeek October 17 launch comparison; evaluation date unpublished. Source names this benchmark/condition but does not restate harness version, task subset or effort. Model-card text confirmed against the official release figure. No September-report scores transferred to the 1017 checkpoint.

60.5% · raw 60.5 %

Alternative · unknown effort · Peer vendor

Source/record date: 2026-10-19

Harness: 600 publicly released AutomationBench tasks; benchmark version 1.0.6, main-table harness and effort unspecified. Not the Claude Code effort sweep. As reported by MiniMax for the exact named peer; not an independent measurement. Existing headline/configuration remains preferred when present. Main figure contains no per-model effort labels. Execution date, exact checkpoint SHA, generation budget and sampling settings are unpublished. Source checked 2026-10-19.

35.4% · raw 35.4 %

Headline · unknown effort · Own vendor

Source/record date: 2026-10-17

Harness: DeepSeek October 17 launch comparison; evaluation date unpublished. Source names this benchmark/condition but does not restate harness version, task subset or effort. Model-card text confirmed against the official release figure. No September-report scores transferred to the 1017 checkpoint.

72.1% · raw 72.1 %

Headline · unknown effort · Own vendor

Source/record date: 2026-10-17

Harness: DeepSeek October 17 launch comparison; evaluation date unpublished. Exact source label Apex Bench v1.1; task population, aggregation and harness not restated. Model-card text confirmed against the official release figure. No September-report scores transferred to the 1017 checkpoint.

83% · raw 83 %

Headline · unknown effort · Own vendor

Source/record date: 2026-10-17

Harness: DeepSeek October 17 launch comparison; evaluation date unpublished. Source names this benchmark/condition but does not restate harness version, task subset or effort. Model-card text confirmed against the official release figure. No September-report scores transferred to the 1017 checkpoint.

92.5% · raw 92.5 %

Headline · unknown effort · Own vendor

Source/record date: 2026-10-17

Harness: DeepSeek October 17 launch comparison; evaluation date unpublished. Source names this benchmark/condition but does not restate harness version, task subset or effort. Model-card text confirmed against the official release figure. No September-report scores transferred to the 1017 checkpoint.

92.5% · raw 92.5 %

Alternative · unknown effort · Peer vendor

Source/record date: 2026-10-19

Harness: Claude Code 2.1.285; minimum image side 1,500 pixels; tools enabled. As reported by MiniMax for the exact named peer; not an independent measurement. Existing headline/configuration remains preferred when present. Main figure contains no per-model effort labels. Execution date, exact checkpoint SHA, generation budget and sampling settings are unpublished. Source checked 2026-10-19.

51% · raw 51 %

Headline · unknown effort · Own vendor

Source/record date: 2026-10-17

Harness: DeepSeek October 17 launch comparison; evaluation date unpublished. ZeroBench-main with tools, pass@5 explicitly stated in launch chart footnote and changelog. Model-card text confirmed against the official release figure. No September-report scores transferred to the 1017 checkpoint.

42.3% · raw 42.3 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-19

Harness: Codex 0.149.1. As reported by MiniMax for the exact named peer; not an independent measurement. Existing headline/configuration remains preferred when present. Main figure contains no per-model effort labels. Execution date, exact checkpoint SHA, generation budget and sampling settings are unpublished. Source checked 2026-10-19.

77% · raw 77 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-19

Harness: mini-swe-agent 2.4.5; maximum context 512K. As reported by MiniMax for the exact named peer; not an independent measurement. Existing headline/configuration remains preferred when present. Main figure contains no per-model effort labels. Execution date, exact checkpoint SHA, generation budget and sampling settings are unpublished. Source checked 2026-10-19.

68.2% · raw 68.2 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-19

Harness: Claude Code 2.1.233; maximum context 512K; reward hacking assigned zero; two task specification/test inconsistencies corrected. As reported by MiniMax for the exact named peer; not an independent measurement. Existing headline/configuration remains preferred when present. Main figure contains no per-model effort labels. Execution date, exact checkpoint SHA, generation budget and sampling settings are unpublished. Source checked 2026-10-19.

61.257% · raw 61.257 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. GDP-weighted aggregate with revised Finance v2.1, EMB v1.1, HLAB v1.1, Code Migration v1.1 and all-pass Tax. Old Index versions are not comparable. Source default includes provider-refusal fallback tasks for Opus/Sonnet; failure-controlled scores are separate. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":2394.709,"stderr":1.055,"cost_per_test":0.65747,"token_totals":{"input_tokens":27106524900,"output_tokens":324131672,"reasoning_tokens":251666265,"cache_read_tokens":26546819188,"cache_write_tokens":null},"temperature":1.0,"top_p":1.0,"max_output_tokens":384000,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"DeepSeek","harness":null}

$0.65747/task · raw 0.65747 USD/task

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. GDP-weighted aggregate with revised Finance v2.1, EMB v1.1, HLAB v1.1, Code Migration v1.1 and all-pass Tax. Old Index versions are not comparable. Source default includes provider-refusal fallback tasks for Opus/Sonnet; failure-controlled scores are separate. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":2394.709,"stderr":1.055,"cost_per_test":0.65747,"token_totals":{"input_tokens":27106524900,"output_tokens":324131672,"reasoning_tokens":251666265,"cache_read_tokens":26546819188,"cache_write_tokens":null},"temperature":1.0,"top_p":1.0,"max_output_tokens":384000,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"DeepSeek","harness":null} Source score standard error describes the benchmark quality score, not uncertainty in cost or latency.

2394.709 s · raw 2394.709 s

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. GDP-weighted aggregate with revised Finance v2.1, EMB v1.1, HLAB v1.1, Code Migration v1.1 and all-pass Tax. Old Index versions are not comparable. Source default includes provider-refusal fallback tasks for Opus/Sonnet; failure-controlled scores are separate. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":2394.709,"stderr":1.055,"cost_per_test":0.65747,"token_totals":{"input_tokens":27106524900,"output_tokens":324131672,"reasoning_tokens":251666265,"cache_read_tokens":26546819188,"cache_write_tokens":null},"temperature":1.0,"top_p":1.0,"max_output_tokens":384000,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"DeepSeek","harness":null} Source score standard error describes the benchmark quality score, not uncertainty in cost or latency.

87.730% · raw 87.73 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: OpenHands. GDP-weighted aggregate with revised Finance v2.1, EMB v1.1, HLAB v1.1, Code Migration v1.1 and all-pass Tax. Old Index versions are not comparable. Source default includes provider-refusal fallback tasks for Opus/Sonnet; failure-controlled scores are separate. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":973.834,"stderr":2.752,"cost_per_test":0.579984,"token_totals":{"input_tokens":2173326299,"output_tokens":9246207,"reasoning_tokens":3107112,"cache_read_tokens":2156782720,"cache_write_tokens":null},"temperature":1.0,"top_p":1.0,"max_output_tokens":384000,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"DeepSeek","harness":"OpenHands"}

62.933% · raw 62.933 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. GDP-weighted aggregate with revised Finance v2.1, EMB v1.1, HLAB v1.1, Code Migration v1.1 and all-pass Tax. Old Index versions are not comparable. Source default includes provider-refusal fallback tasks for Opus/Sonnet; failure-controlled scores are separate. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":909.212,"stderr":2.028,"cost_per_test":0.466261,"token_totals":{"input_tokens":1025601275,"output_tokens":94944123,"reasoning_tokens":82978352,"cache_read_tokens":721979776,"cache_write_tokens":null},"temperature":1.0,"top_p":1.0,"max_output_tokens":384000,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"DeepSeek","harness":null}

69.173% · raw 69.173 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. GDP-weighted aggregate with revised Finance v2.1, EMB v1.1, HLAB v1.1, Code Migration v1.1 and all-pass Tax. Old Index versions are not comparable. Source default includes provider-refusal fallback tasks for Opus/Sonnet; failure-controlled scores are separate. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":1971.687,"stderr":2.395,"cost_per_test":0.330922,"token_totals":{"input_tokens":1809205546,"output_tokens":16117195,"reasoning_tokens":11510477,"cache_read_tokens":1797102837,"cache_write_tokens":null},"temperature":1.0,"top_p":1.0,"max_output_tokens":384000,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"DeepSeek","harness":null}

68.500% · raw 68.5 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. GDP-weighted aggregate with revised Finance v2.1, EMB v1.1, HLAB v1.1, Code Migration v1.1 and all-pass Tax. Old Index versions are not comparable. Source default includes provider-refusal fallback tasks for Opus/Sonnet; failure-controlled scores are separate. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":8133.069,"stderr":3.476,"cost_per_test":1.572628,"token_totals":{"input_tokens":11188739656,"output_tokens":22875100,"reasoning_tokens":10958121,"cache_read_tokens":11164665216,"cache_write_tokens":0},"temperature":1.0,"top_p":1.0,"max_output_tokens":384000,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"DeepSeek","harness":null}

45.192% · raw 45.192 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. GDP-weighted aggregate with revised Finance v2.1, EMB v1.1, HLAB v1.1, Code Migration v1.1 and all-pass Tax. Old Index versions are not comparable. Source default includes provider-refusal fallback tasks for Opus/Sonnet; failure-controlled scores are separate. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":2106.881,"stderr":3.459,"cost_per_test":0.69856,"token_totals":{"input_tokens":2003160078,"output_tokens":87441264,"reasoning_tokens":76652436,"cache_read_tokens":1909100928,"cache_write_tokens":0},"temperature":1.0,"top_p":1.0,"max_output_tokens":384000,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"DeepSeek","harness":null}

14.167% · raw 14.167 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. GDP-weighted aggregate with revised Finance v2.1, EMB v1.1, HLAB v1.1, Code Migration v1.1 and all-pass Tax. Old Index versions are not comparable. Source default includes provider-refusal fallback tasks for Opus/Sonnet; failure-controlled scores are separate. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":1328.108,"stderr":2.787,"cost_per_test":0.612596,"token_totals":{"input_tokens":5524481146,"output_tokens":25881985,"reasoning_tokens":10520362,"cache_read_tokens":5492827392,"cache_write_tokens":null},"temperature":1.0,"top_p":1.0,"max_output_tokens":384000,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"DeepSeek","harness":null}

31.818% · raw 31.818 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. GDP-weighted aggregate with revised Finance v2.1, EMB v1.1, HLAB v1.1, Code Migration v1.1 and all-pass Tax. Old Index versions are not comparable. Source default includes provider-refusal fallback tasks for Opus/Sonnet; failure-controlled scores are separate. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":2470.449,"stderr":2.624,"cost_per_test":0.539359,"token_totals":{"input_tokens":2602850452,"output_tokens":14064073,"reasoning_tokens":10223328,"cache_read_tokens":2592293631,"cache_write_tokens":0},"temperature":1.0,"top_p":1.0,"max_output_tokens":384000,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"DeepSeek","harness":null}

44.350% · raw 44.35 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. GDP-weighted aggregate with revised Finance v2.1, EMB v1.1, HLAB v1.1, Code Migration v1.1 and all-pass Tax. Old Index versions are not comparable. Source default includes provider-refusal fallback tasks for Opus/Sonnet; failure-controlled scores are separate. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":1264.434,"stderr":3.574,"cost_per_test":0.459454,"token_totals":{"input_tokens":779160448,"output_tokens":53561725,"reasoning_tokens":45716077,"cache_read_tokens":712066688,"cache_write_tokens":null},"temperature":1.0,"top_p":1.0,"max_output_tokens":384000,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"DeepSeek","harness":null}

62.933% · raw 62.933 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. FAB v2 harness on expert-reviewed FABv2.1 questions/rubrics, revised judge jury and hedged-answer prompt. Partial-credit accuracy differs from all-pass. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":909.212,"stderr":2.028,"cost_per_test":0.466261,"token_totals":{"input_tokens":1025601275,"output_tokens":94944123,"reasoning_tokens":82978352,"cache_read_tokens":721979776,"cache_write_tokens":null},"temperature":1.0,"top_p":1.0,"max_output_tokens":384000,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"DeepSeek","harness":null}

$0.466261/task · raw 0.466261 USD/task

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. FAB v2 harness on expert-reviewed FABv2.1 questions/rubrics, revised judge jury and hedged-answer prompt. Partial-credit accuracy differs from all-pass. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":909.212,"stderr":2.028,"cost_per_test":0.466261,"token_totals":{"input_tokens":1025601275,"output_tokens":94944123,"reasoning_tokens":82978352,"cache_read_tokens":721979776,"cache_write_tokens":null},"temperature":1.0,"top_p":1.0,"max_output_tokens":384000,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"DeepSeek","harness":null} Source score standard error describes the benchmark quality score, not uncertainty in cost or latency.

909.212 s · raw 909.212 s

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. FAB v2 harness on expert-reviewed FABv2.1 questions/rubrics, revised judge jury and hedged-answer prompt. Partial-credit accuracy differs from all-pass. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":909.212,"stderr":2.028,"cost_per_test":0.466261,"token_totals":{"input_tokens":1025601275,"output_tokens":94944123,"reasoning_tokens":82978352,"cache_read_tokens":721979776,"cache_write_tokens":null},"temperature":1.0,"top_p":1.0,"max_output_tokens":384000,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"DeepSeek","harness":null} Source score standard error describes the benchmark quality score, not uncertainty in cost or latency.

51.345% · raw 51.345 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. FAB v2 harness on expert-reviewed FABv2.1 questions/rubrics, revised judge jury and hedged-answer prompt. Partial-credit accuracy differs from all-pass. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":909.212,"stderr":2.198,"cost_per_test":0.466261,"token_totals":{"input_tokens":1025601275,"output_tokens":94944123,"reasoning_tokens":82978352,"cache_read_tokens":721979776,"cache_write_tokens":null},"temperature":1.0,"top_p":1.0,"max_output_tokens":384000,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"DeepSeek","harness":null}

70.995% · raw 70.995 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. FAB v2 harness on expert-reviewed FABv2.1 questions/rubrics, revised judge jury and hedged-answer prompt. Partial-credit accuracy differs from all-pass. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":1143.475,"stderr":5.818,"cost_per_test":0.573931,"temperature":1.0,"top_p":1.0,"max_output_tokens":384000,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"DeepSeek","harness":null}

75.971% · raw 75.971 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. FAB v2 harness on expert-reviewed FABv2.1 questions/rubrics, revised judge jury and hedged-answer prompt. Partial-credit accuracy differs from all-pass. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":720.164,"stderr":5.258,"cost_per_test":0.435356,"temperature":1.0,"top_p":1.0,"max_output_tokens":384000,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"DeepSeek","harness":null}

60.690% · raw 60.69 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. FAB v2 harness on expert-reviewed FABv2.1 questions/rubrics, revised judge jury and hedged-answer prompt. Partial-credit accuracy differs from all-pass. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":637.261,"stderr":6.951,"cost_per_test":0.286289,"temperature":1.0,"top_p":1.0,"max_output_tokens":384000,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"DeepSeek","harness":null}

59.751% · raw 59.751 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. FAB v2 harness on expert-reviewed FABv2.1 questions/rubrics, revised judge jury and hedged-answer prompt. Partial-credit accuracy differs from all-pass. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":1064.803,"stderr":6.375,"cost_per_test":0.51929,"temperature":1.0,"top_p":1.0,"max_output_tokens":384000,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"DeepSeek","harness":null}

29.348% · raw 29.348 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. FAB v2 harness on expert-reviewed FABv2.1 questions/rubrics, revised judge jury and hedged-answer prompt. Partial-credit accuracy differs from all-pass. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":1361.053,"stderr":6.67,"cost_per_test":0.709077,"temperature":1.0,"top_p":1.0,"max_output_tokens":384000,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"DeepSeek","harness":null}

66.771% · raw 66.771 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. FAB v2 harness on expert-reviewed FABv2.1 questions/rubrics, revised judge jury and hedged-answer prompt. Partial-credit accuracy differs from all-pass. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":954.029,"stderr":6.591,"cost_per_test":0.506283,"temperature":1.0,"top_p":1.0,"max_output_tokens":384000,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"DeepSeek","harness":null}

93.029% · raw 93.029 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. FAB v2 harness on expert-reviewed FABv2.1 questions/rubrics, revised judge jury and hedged-answer prompt. Partial-credit accuracy differs from all-pass. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":565.155,"stderr":3.411,"cost_per_test":0.22062,"temperature":1.0,"top_p":1.0,"max_output_tokens":384000,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"DeepSeek","harness":null}

70.262% · raw 70.262 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. FAB v2 harness on expert-reviewed FABv2.1 questions/rubrics, revised judge jury and hedged-answer prompt. Partial-credit accuracy differs from all-pass. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":1178.026,"stderr":6.153,"cost_per_test":0.654078,"temperature":1.0,"top_p":1.0,"max_output_tokens":384000,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"DeepSeek","harness":null}

39.583% · raw 39.583 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. FAB v2 harness on expert-reviewed FABv2.1 questions/rubrics, revised judge jury and hedged-answer prompt. Partial-credit accuracy differs from all-pass. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":668.471,"stderr":6.73,"cost_per_test":0.318493,"temperature":1.0,"top_p":1.0,"max_output_tokens":384000,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"DeepSeek","harness":null}

69.173% · raw 69.173 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Seven equally weighted category scores combining Template and Scratch. GPT-6 Luna Scratch judge and Template sheet-name scoring fixes; not comparable with v1.0. Partial credit is not client-ready quality. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":1971.687,"stderr":2.395,"cost_per_test":0.330922,"token_totals":{"input_tokens":1809205546,"output_tokens":16117195,"reasoning_tokens":11510477,"cache_read_tokens":1797102837,"cache_write_tokens":null},"temperature":1.0,"top_p":1.0,"max_output_tokens":384000,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"DeepSeek","harness":null}

$0.330922/task · raw 0.330922 USD/task

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Seven equally weighted category scores combining Template and Scratch. GPT-6 Luna Scratch judge and Template sheet-name scoring fixes; not comparable with v1.0. Partial credit is not client-ready quality. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":1971.687,"stderr":2.395,"cost_per_test":0.330922,"token_totals":{"input_tokens":1809205546,"output_tokens":16117195,"reasoning_tokens":11510477,"cache_read_tokens":1797102837,"cache_write_tokens":null},"temperature":1.0,"top_p":1.0,"max_output_tokens":384000,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"DeepSeek","harness":null} Source score standard error describes the benchmark quality score, not uncertainty in cost or latency.

1971.687 s · raw 1971.687 s

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Seven equally weighted category scores combining Template and Scratch. GPT-6 Luna Scratch judge and Template sheet-name scoring fixes; not comparable with v1.0. Partial credit is not client-ready quality. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":1971.687,"stderr":2.395,"cost_per_test":0.330922,"token_totals":{"input_tokens":1809205546,"output_tokens":16117195,"reasoning_tokens":11510477,"cache_read_tokens":1797102837,"cache_write_tokens":null},"temperature":1.0,"top_p":1.0,"max_output_tokens":384000,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"DeepSeek","harness":null} Source score standard error describes the benchmark quality score, not uncertainty in cost or latency.

70.829% · raw 70.829 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Seven equally weighted category scores combining Template and Scratch. GPT-6 Luna Scratch judge and Template sheet-name scoring fixes; not comparable with v1.0. Partial credit is not client-ready quality. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":1913.359,"stderr":2.907,"cost_per_test":0.346279,"temperature":1.0,"top_p":1.0,"max_output_tokens":384000,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"DeepSeek","harness":null}

67.325% · raw 67.325 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Seven equally weighted category scores combining Template and Scratch. GPT-6 Luna Scratch judge and Template sheet-name scoring fixes; not comparable with v1.0. Partial credit is not client-ready quality. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":2030.015,"stderr":2.64,"cost_per_test":0.315566,"temperature":1.0,"top_p":1.0,"max_output_tokens":384000,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"DeepSeek","harness":null}

62.463% · raw 62.463 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Seven equally weighted category scores combining Template and Scratch. GPT-6 Luna Scratch judge and Template sheet-name scoring fixes; not comparable with v1.0. Partial credit is not client-ready quality. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":4170.026,"stderr":4.415,"cost_per_test":0.419172,"temperature":1.0,"top_p":1.0,"max_output_tokens":384000,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"DeepSeek","harness":null}

59.710% · raw 59.71 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Seven equally weighted category scores combining Template and Scratch. GPT-6 Luna Scratch judge and Template sheet-name scoring fixes; not comparable with v1.0. Partial credit is not client-ready quality. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":3628.584,"stderr":7.04,"cost_per_test":0.336662,"temperature":1.0,"top_p":1.0,"max_output_tokens":384000,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"DeepSeek","harness":null}

73.081% · raw 73.081 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Seven equally weighted category scores combining Template and Scratch. GPT-6 Luna Scratch judge and Template sheet-name scoring fixes; not comparable with v1.0. Partial credit is not client-ready quality. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":2347.799,"stderr":5.038,"cost_per_test":0.286688,"temperature":1.0,"top_p":1.0,"max_output_tokens":384000,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"DeepSeek","harness":null}

65.965% · raw 65.965 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Seven equally weighted category scores combining Template and Scratch. GPT-6 Luna Scratch judge and Template sheet-name scoring fixes; not comparable with v1.0. Partial credit is not client-ready quality. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":842.327,"stderr":4.694,"cost_per_test":0.28596,"temperature":1.0,"top_p":1.0,"max_output_tokens":384000,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"DeepSeek","harness":null}

70.771% · raw 70.771 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Seven equally weighted category scores combining Template and Scratch. GPT-6 Luna Scratch judge and Template sheet-name scoring fixes; not comparable with v1.0. Partial credit is not client-ready quality. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":790.726,"stderr":4.335,"cost_per_test":0.331686,"temperature":1.0,"top_p":1.0,"max_output_tokens":384000,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"DeepSeek","harness":null}

86.495% · raw 86.495 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Seven equally weighted category scores combining Template and Scratch. GPT-6 Luna Scratch judge and Template sheet-name scoring fixes; not comparable with v1.0. Partial credit is not client-ready quality. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":919.03,"stderr":4.278,"cost_per_test":0.315607,"temperature":1.0,"top_p":1.0,"max_output_tokens":384000,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"DeepSeek","harness":null}

65.723% · raw 65.723 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Seven equally weighted category scores combining Template and Scratch. GPT-6 Luna Scratch judge and Template sheet-name scoring fixes; not comparable with v1.0. Partial credit is not client-ready quality. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":977.252,"stderr":3.32,"cost_per_test":0.334964,"temperature":1.0,"top_p":1.0,"max_output_tokens":384000,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"DeepSeek","harness":null}

66.599% · raw 66.599 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Seven equally weighted category scores combining Template and Scratch. GPT-6 Luna Scratch judge and Template sheet-name scoring fixes; not comparable with v1.0. Partial credit is not client-ready quality. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":1971.687,"stderr":2.395,"cost_per_test":0.330922,"token_totals":{"input_tokens":1809205546,"output_tokens":16117195,"reasoning_tokens":11510477,"cache_read_tokens":1797102837,"cache_write_tokens":null},"temperature":1.0,"top_p":1.0,"max_output_tokens":384000,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"DeepSeek","harness":null}

58.327% · raw 58.327 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Seven equally weighted category scores combining Template and Scratch. GPT-6 Luna Scratch judge and Template sheet-name scoring fixes; not comparable with v1.0. Partial credit is not client-ready quality. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":1971.687,"stderr":2.395,"cost_per_test":0.330922,"token_totals":{"input_tokens":1809205546,"output_tokens":16117195,"reasoning_tokens":11510477,"cache_read_tokens":1797102837,"cache_write_tokens":null},"temperature":1.0,"top_p":1.0,"max_output_tokens":384000,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"DeepSeek","harness":null}

59.865% · raw 59.865 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Seven equally weighted category scores combining Template and Scratch. GPT-6 Luna Scratch judge and Template sheet-name scoring fixes; not comparable with v1.0. Partial credit is not client-ready quality. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":1971.687,"stderr":2.395,"cost_per_test":0.330922,"token_totals":{"input_tokens":1809205546,"output_tokens":16117195,"reasoning_tokens":11510477,"cache_read_tokens":1797102837,"cache_write_tokens":null},"temperature":1.0,"top_p":1.0,"max_output_tokens":384000,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"DeepSeek","harness":null}

59.555% · raw 59.555 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Seven equally weighted category scores combining Template and Scratch. GPT-6 Luna Scratch judge and Template sheet-name scoring fixes; not comparable with v1.0. Partial credit is not client-ready quality. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":1971.687,"stderr":2.395,"cost_per_test":0.330922,"token_totals":{"input_tokens":1809205546,"output_tokens":16117195,"reasoning_tokens":11510477,"cache_read_tokens":1797102837,"cache_write_tokens":null},"temperature":1.0,"top_p":1.0,"max_output_tokens":384000,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"DeepSeek","harness":null}

68.686% · raw 68.686 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Seven equally weighted category scores combining Template and Scratch. GPT-6 Luna Scratch judge and Template sheet-name scoring fixes; not comparable with v1.0. Partial credit is not client-ready quality. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":1971.687,"stderr":2.395,"cost_per_test":0.330922,"token_totals":{"input_tokens":1809205546,"output_tokens":16117195,"reasoning_tokens":11510477,"cache_read_tokens":1797102837,"cache_write_tokens":null},"temperature":1.0,"top_p":1.0,"max_output_tokens":384000,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"DeepSeek","harness":null}

77.475% · raw 77.475 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Seven equally weighted category scores combining Template and Scratch. GPT-6 Luna Scratch judge and Template sheet-name scoring fixes; not comparable with v1.0. Partial credit is not client-ready quality. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":1971.687,"stderr":2.395,"cost_per_test":0.330922,"token_totals":{"input_tokens":1809205546,"output_tokens":16117195,"reasoning_tokens":11510477,"cache_read_tokens":1797102837,"cache_write_tokens":null},"temperature":1.0,"top_p":1.0,"max_output_tokens":384000,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"DeepSeek","harness":null}

70.625% · raw 70.625 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Seven equally weighted category scores combining Template and Scratch. GPT-6 Luna Scratch judge and Template sheet-name scoring fixes; not comparable with v1.0. Partial credit is not client-ready quality. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":1971.687,"stderr":2.395,"cost_per_test":0.330922,"token_totals":{"input_tokens":1809205546,"output_tokens":16117195,"reasoning_tokens":11510477,"cache_read_tokens":1797102837,"cache_write_tokens":null},"temperature":1.0,"top_p":1.0,"max_output_tokens":384000,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"DeepSeek","harness":null}

61.305% · raw 61.305 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Seven equally weighted category scores combining Template and Scratch. GPT-6 Luna Scratch judge and Template sheet-name scoring fixes; not comparable with v1.0. Partial credit is not client-ready quality. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":1971.687,"stderr":2.395,"cost_per_test":0.330922,"token_totals":{"input_tokens":1809205546,"output_tokens":16117195,"reasoning_tokens":11510477,"cache_read_tokens":1797102837,"cache_write_tokens":null},"temperature":1.0,"top_p":1.0,"max_output_tokens":384000,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"DeepSeek","harness":null}

71.289% · raw 71.289 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Seven equally weighted category scores combining Template and Scratch. GPT-6 Luna Scratch judge and Template sheet-name scoring fixes; not comparable with v1.0. Partial credit is not client-ready quality. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":1971.687,"stderr":2.395,"cost_per_test":0.330922,"token_totals":{"input_tokens":1809205546,"output_tokens":16117195,"reasoning_tokens":11510477,"cache_read_tokens":1797102837,"cache_write_tokens":null},"temperature":1.0,"top_p":1.0,"max_output_tokens":384000,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"DeepSeek","harness":null}

70.254% · raw 70.254 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Seven equally weighted category scores combining Template and Scratch. GPT-6 Luna Scratch judge and Template sheet-name scoring fixes; not comparable with v1.0. Partial credit is not client-ready quality. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":1971.687,"stderr":2.395,"cost_per_test":0.330922,"token_totals":{"input_tokens":1809205546,"output_tokens":16117195,"reasoning_tokens":11510477,"cache_read_tokens":1797102837,"cache_write_tokens":null},"temperature":1.0,"top_p":1.0,"max_output_tokens":384000,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"DeepSeek","harness":null}

91.533% · raw 91.533 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Seven equally weighted category scores combining Template and Scratch. GPT-6 Luna Scratch judge and Template sheet-name scoring fixes; not comparable with v1.0. Partial credit is not client-ready quality. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":1971.687,"stderr":2.395,"cost_per_test":0.330922,"token_totals":{"input_tokens":1809205546,"output_tokens":16117195,"reasoning_tokens":11510477,"cache_read_tokens":1797102837,"cache_write_tokens":null},"temperature":1.0,"top_p":1.0,"max_output_tokens":384000,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"DeepSeek","harness":null}

81.457% · raw 81.457 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Seven equally weighted category scores combining Template and Scratch. GPT-6 Luna Scratch judge and Template sheet-name scoring fixes; not comparable with v1.0. Partial credit is not client-ready quality. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":1971.687,"stderr":2.395,"cost_per_test":0.330922,"token_totals":{"input_tokens":1809205546,"output_tokens":16117195,"reasoning_tokens":11510477,"cache_read_tokens":1797102837,"cache_write_tokens":null},"temperature":1.0,"top_p":1.0,"max_output_tokens":384000,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"DeepSeek","harness":null}

67.264% · raw 67.264 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Seven equally weighted category scores combining Template and Scratch. GPT-6 Luna Scratch judge and Template sheet-name scoring fixes; not comparable with v1.0. Partial credit is not client-ready quality. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":1971.687,"stderr":2.395,"cost_per_test":0.330922,"token_totals":{"input_tokens":1809205546,"output_tokens":16117195,"reasoning_tokens":11510477,"cache_read_tokens":1797102837,"cache_write_tokens":null},"temperature":1.0,"top_p":1.0,"max_output_tokens":384000,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"DeepSeek","harness":null}

64.182% · raw 64.182 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Seven equally weighted category scores combining Template and Scratch. GPT-6 Luna Scratch judge and Template sheet-name scoring fixes; not comparable with v1.0. Partial credit is not client-ready quality. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":1971.687,"stderr":2.395,"cost_per_test":0.330922,"token_totals":{"input_tokens":1809205546,"output_tokens":16117195,"reasoning_tokens":11510477,"cache_read_tokens":1797102837,"cache_write_tokens":null},"temperature":1.0,"top_p":1.0,"max_output_tokens":384000,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"DeepSeek","harness":null}

59.617% · raw 59.617 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Seven equally weighted category scores combining Template and Scratch. GPT-6 Luna Scratch judge and Template sheet-name scoring fixes; not comparable with v1.0. Partial credit is not client-ready quality. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":1971.687,"stderr":2.395,"cost_per_test":0.330922,"token_totals":{"input_tokens":1809205546,"output_tokens":16117195,"reasoning_tokens":11510477,"cache_read_tokens":1797102837,"cache_write_tokens":null},"temperature":1.0,"top_p":1.0,"max_output_tokens":384000,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"DeepSeek","harness":null}

86.671% · raw 86.671 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Seven equally weighted category scores combining Template and Scratch. GPT-6 Luna Scratch judge and Template sheet-name scoring fixes; not comparable with v1.0. Partial credit is not client-ready quality. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":1971.687,"stderr":2.395,"cost_per_test":0.330922,"token_totals":{"input_tokens":1809205546,"output_tokens":16117195,"reasoning_tokens":11510477,"cache_read_tokens":1797102837,"cache_write_tokens":null},"temperature":1.0,"top_p":1.0,"max_output_tokens":384000,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"DeepSeek","harness":null}

73.832% · raw 73.832 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Seven equally weighted category scores combining Template and Scratch. GPT-6 Luna Scratch judge and Template sheet-name scoring fixes; not comparable with v1.0. Partial credit is not client-ready quality. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":1971.687,"stderr":2.395,"cost_per_test":0.330922,"token_totals":{"input_tokens":1809205546,"output_tokens":16117195,"reasoning_tokens":11510477,"cache_read_tokens":1797102837,"cache_write_tokens":null},"temperature":1.0,"top_p":1.0,"max_output_tokens":384000,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"DeepSeek","harness":null}

67.524% · raw 67.524 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: Pi. Pi agent harness, revised hidden tests, full 120 CLI + 10 COBOL tasks; 75% CLI and 25% COBOL weighting. Distinct from Index selected 50 CLI +10 COBOL subset and old mini-SWE-agent v1. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":8475.469,"stderr":2.763,"cost_per_test":1.764297,"token_totals":{"input_tokens":29616647741,"output_tokens":57114571,"reasoning_tokens":26518612,"cache_read_tokens":29542723456,"cache_write_tokens":0},"temperature":1.0,"top_p":1.0,"max_output_tokens":384000,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"DeepSeek","harness":null}

$1.764297/task · raw 1.764297 USD/task

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: Pi. Pi agent harness, revised hidden tests, full 120 CLI + 10 COBOL tasks; 75% CLI and 25% COBOL weighting. Distinct from Index selected 50 CLI +10 COBOL subset and old mini-SWE-agent v1. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":8475.469,"stderr":2.763,"cost_per_test":1.764297,"token_totals":{"input_tokens":29616647741,"output_tokens":57114571,"reasoning_tokens":26518612,"cache_read_tokens":29542723456,"cache_write_tokens":0},"temperature":1.0,"top_p":1.0,"max_output_tokens":384000,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"DeepSeek","harness":null} Source score standard error describes the benchmark quality score, not uncertainty in cost or latency.

8475.469 s · raw 8475.469 s

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: Pi. Pi agent harness, revised hidden tests, full 120 CLI + 10 COBOL tasks; 75% CLI and 25% COBOL weighting. Distinct from Index selected 50 CLI +10 COBOL subset and old mini-SWE-agent v1. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":8475.469,"stderr":2.763,"cost_per_test":1.764297,"token_totals":{"input_tokens":29616647741,"output_tokens":57114571,"reasoning_tokens":26518612,"cache_read_tokens":29542723456,"cache_write_tokens":0},"temperature":1.0,"top_p":1.0,"max_output_tokens":384000,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"DeepSeek","harness":null} Source score standard error describes the benchmark quality score, not uncertainty in cost or latency.

59.199% · raw 59.199 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: Pi. Pi agent harness, revised hidden tests, full 120 CLI + 10 COBOL tasks; 75% CLI and 25% COBOL weighting. Distinct from Index selected 50 CLI +10 COBOL subset and old mini-SWE-agent v1. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":10029.729,"stderr":2.861,"cost_per_test":2.193325,"temperature":1.0,"top_p":1.0,"max_output_tokens":384000,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"DeepSeek","harness":null}

92.500% · raw 92.5 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: Pi. Pi agent harness, revised hidden tests, full 120 CLI + 10 COBOL tasks; 75% CLI and 25% COBOL weighting. Distinct from Index selected 50 CLI +10 COBOL subset and old mini-SWE-agent v1. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":3812.688,"stderr":6.962,"cost_per_test":0.477214,"temperature":1.0,"top_p":1.0,"max_output_tokens":384000,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"DeepSeek","harness":null}

3.476 points/5 · raw 3.476 points/5

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: Pi. Pi agent harness, revised hidden tests, full 120 CLI + 10 COBOL tasks; 75% CLI and 25% COBOL weighting. Distinct from Index selected 50 CLI +10 COBOL subset and old mini-SWE-agent v1. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":8475.469,"stderr":0.033,"cost_per_test":1.764297,"temperature":1.0,"top_p":1.0,"max_output_tokens":384000,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"DeepSeek","harness":null}

14.167% · raw 14.167 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Harvey held-out legal work protocol, internet disabled, GPT5.5 medium and Sonnet4.6 judges. Task pass requires every criterion; final is mean of two judges. Criteria-pass rate is separate. DOCX tracked-change grading fix. Distinct from AA hallucination-gated LAB. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":1328.108,"stderr":2.787,"cost_per_test":0.612596,"token_totals":{"input_tokens":5524481146,"output_tokens":25881985,"reasoning_tokens":10520362,"cache_read_tokens":5492827392,"cache_write_tokens":null},"temperature":1.0,"top_p":1.0,"max_output_tokens":384000,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"DeepSeek","harness":null}

$0.612596/task · raw 0.612596 USD/task

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Harvey held-out legal work protocol, internet disabled, GPT5.5 medium and Sonnet4.6 judges. Task pass requires every criterion; final is mean of two judges. Criteria-pass rate is separate. DOCX tracked-change grading fix. Distinct from AA hallucination-gated LAB. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":1328.108,"stderr":2.787,"cost_per_test":0.612596,"token_totals":{"input_tokens":5524481146,"output_tokens":25881985,"reasoning_tokens":10520362,"cache_read_tokens":5492827392,"cache_write_tokens":null},"temperature":1.0,"top_p":1.0,"max_output_tokens":384000,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"DeepSeek","harness":null} Source score standard error describes the benchmark quality score, not uncertainty in cost or latency.

1328.108 s · raw 1328.108 s

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Harvey held-out legal work protocol, internet disabled, GPT5.5 medium and Sonnet4.6 judges. Task pass requires every criterion; final is mean of two judges. Criteria-pass rate is separate. DOCX tracked-change grading fix. Distinct from AA hallucination-gated LAB. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":1328.108,"stderr":2.787,"cost_per_test":0.612596,"token_totals":{"input_tokens":5524481146,"output_tokens":25881985,"reasoning_tokens":10520362,"cache_read_tokens":5492827392,"cache_write_tokens":null},"temperature":1.0,"top_p":1.0,"max_output_tokens":384000,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"DeepSeek","harness":null} Source score standard error describes the benchmark quality score, not uncertainty in cost or latency.

30.000% · raw 30 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Harvey held-out legal work protocol, internet disabled, GPT5.5 medium and Sonnet4.6 judges. Task pass requires every criterion; final is mean of two judges. Criteria-pass rate is separate. DOCX tracked-change grading fix. Distinct from AA hallucination-gated LAB. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":1264.71,"stderr":20.0,"cost_per_test":0.534289,"temperature":1.0,"top_p":1.0,"max_output_tokens":384000,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"DeepSeek","harness":null}

10.000% · raw 10 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Harvey held-out legal work protocol, internet disabled, GPT5.5 medium and Sonnet4.6 judges. Task pass requires every criterion; final is mean of two judges. Criteria-pass rate is separate. DOCX tracked-change grading fix. Distinct from AA hallucination-gated LAB. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":1273.796,"stderr":10.0,"cost_per_test":0.66721,"temperature":1.0,"top_p":1.0,"max_output_tokens":384000,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"DeepSeek","harness":null}

10.000% · raw 10 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Harvey held-out legal work protocol, internet disabled, GPT5.5 medium and Sonnet4.6 judges. Task pass requires every criterion; final is mean of two judges. Criteria-pass rate is separate. DOCX tracked-change grading fix. Distinct from AA hallucination-gated LAB. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":1273.962,"stderr":10.0,"cost_per_test":0.604563,"temperature":1.0,"top_p":1.0,"max_output_tokens":384000,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"DeepSeek","harness":null}

10.000% · raw 10 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Harvey held-out legal work protocol, internet disabled, GPT5.5 medium and Sonnet4.6 judges. Task pass requires every criterion; final is mean of two judges. Criteria-pass rate is separate. DOCX tracked-change grading fix. Distinct from AA hallucination-gated LAB. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":1285.442,"stderr":10.0,"cost_per_test":0.640206,"temperature":1.0,"top_p":1.0,"max_output_tokens":384000,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"DeepSeek","harness":null}

20.000% · raw 20 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Harvey held-out legal work protocol, internet disabled, GPT5.5 medium and Sonnet4.6 judges. Task pass requires every criterion; final is mean of two judges. Criteria-pass rate is separate. DOCX tracked-change grading fix. Distinct from AA hallucination-gated LAB. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":1270.292,"stderr":20.0,"cost_per_test":0.604559,"temperature":1.0,"top_p":1.0,"max_output_tokens":384000,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"DeepSeek","harness":null}

20.000% · raw 20 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Harvey held-out legal work protocol, internet disabled, GPT5.5 medium and Sonnet4.6 judges. Task pass requires every criterion; final is mean of two judges. Criteria-pass rate is separate. DOCX tracked-change grading fix. Distinct from AA hallucination-gated LAB. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":1497.44,"stderr":20.0,"cost_per_test":0.70205,"temperature":1.0,"top_p":1.0,"max_output_tokens":384000,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"DeepSeek","harness":null}

20.000% · raw 20 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Harvey held-out legal work protocol, internet disabled, GPT5.5 medium and Sonnet4.6 judges. Task pass requires every criterion; final is mean of two judges. Criteria-pass rate is separate. DOCX tracked-change grading fix. Distinct from AA hallucination-gated LAB. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":1348.014,"stderr":20.0,"cost_per_test":0.64235,"temperature":1.0,"top_p":1.0,"max_output_tokens":384000,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"DeepSeek","harness":null}

10.000% · raw 10 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Harvey held-out legal work protocol, internet disabled, GPT5.5 medium and Sonnet4.6 judges. Task pass requires every criterion; final is mean of two judges. Criteria-pass rate is separate. DOCX tracked-change grading fix. Distinct from AA hallucination-gated LAB. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":1228.252,"stderr":10.0,"cost_per_test":0.52666,"temperature":1.0,"top_p":1.0,"max_output_tokens":384000,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"DeepSeek","harness":null}

0.000% · raw 0 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Harvey held-out legal work protocol, internet disabled, GPT5.5 medium and Sonnet4.6 judges. Task pass requires every criterion; final is mean of two judges. Criteria-pass rate is separate. DOCX tracked-change grading fix. Distinct from AA hallucination-gated LAB. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":1539.008,"stderr":0.0,"cost_per_test":0.633521,"temperature":1.0,"top_p":1.0,"max_output_tokens":384000,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"DeepSeek","harness":null}

0.000% · raw 0 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Harvey held-out legal work protocol, internet disabled, GPT5.5 medium and Sonnet4.6 judges. Task pass requires every criterion; final is mean of two judges. Criteria-pass rate is separate. DOCX tracked-change grading fix. Distinct from AA hallucination-gated LAB. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":1291.326,"stderr":0.0,"cost_per_test":0.520216,"temperature":1.0,"top_p":1.0,"max_output_tokens":384000,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"DeepSeek","harness":null}

30.000% · raw 30 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Harvey held-out legal work protocol, internet disabled, GPT5.5 medium and Sonnet4.6 judges. Task pass requires every criterion; final is mean of two judges. Criteria-pass rate is separate. DOCX tracked-change grading fix. Distinct from AA hallucination-gated LAB. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":1272.778,"stderr":20.0,"cost_per_test":0.69079,"temperature":1.0,"top_p":1.0,"max_output_tokens":384000,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"DeepSeek","harness":null}

0.000% · raw 0 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Harvey held-out legal work protocol, internet disabled, GPT5.5 medium and Sonnet4.6 judges. Task pass requires every criterion; final is mean of two judges. Criteria-pass rate is separate. DOCX tracked-change grading fix. Distinct from AA hallucination-gated LAB. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":1433.79,"stderr":0.0,"cost_per_test":0.635298,"temperature":1.0,"top_p":1.0,"max_output_tokens":384000,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"DeepSeek","harness":null}

40.000% · raw 40 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Harvey held-out legal work protocol, internet disabled, GPT5.5 medium and Sonnet4.6 judges. Task pass requires every criterion; final is mean of two judges. Criteria-pass rate is separate. DOCX tracked-change grading fix. Distinct from AA hallucination-gated LAB. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":1356.48,"stderr":18.708,"cost_per_test":0.654524,"temperature":1.0,"top_p":1.0,"max_output_tokens":384000,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"DeepSeek","harness":null}

30.000% · raw 30 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Harvey held-out legal work protocol, internet disabled, GPT5.5 medium and Sonnet4.6 judges. Task pass requires every criterion; final is mean of two judges. Criteria-pass rate is separate. DOCX tracked-change grading fix. Distinct from AA hallucination-gated LAB. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":1457.208,"stderr":20.0,"cost_per_test":0.656838,"temperature":1.0,"top_p":1.0,"max_output_tokens":384000,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"DeepSeek","harness":null}

20.000% · raw 20 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Harvey held-out legal work protocol, internet disabled, GPT5.5 medium and Sonnet4.6 judges. Task pass requires every criterion; final is mean of two judges. Criteria-pass rate is separate. DOCX tracked-change grading fix. Distinct from AA hallucination-gated LAB. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":1218.928,"stderr":20.0,"cost_per_test":0.468313,"temperature":1.0,"top_p":1.0,"max_output_tokens":384000,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"DeepSeek","harness":null}

0.000% · raw 0 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Harvey held-out legal work protocol, internet disabled, GPT5.5 medium and Sonnet4.6 judges. Task pass requires every criterion; final is mean of two judges. Criteria-pass rate is separate. DOCX tracked-change grading fix. Distinct from AA hallucination-gated LAB. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":1109.874,"stderr":0.0,"cost_per_test":0.476309,"temperature":1.0,"top_p":1.0,"max_output_tokens":384000,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"DeepSeek","harness":null}

10.000% · raw 10 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Harvey held-out legal work protocol, internet disabled, GPT5.5 medium and Sonnet4.6 judges. Task pass requires every criterion; final is mean of two judges. Criteria-pass rate is separate. DOCX tracked-change grading fix. Distinct from AA hallucination-gated LAB. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":1311.01,"stderr":10.0,"cost_per_test":0.601227,"temperature":1.0,"top_p":1.0,"max_output_tokens":384000,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"DeepSeek","harness":null}

0.000% · raw 0 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Harvey held-out legal work protocol, internet disabled, GPT5.5 medium and Sonnet4.6 judges. Task pass requires every criterion; final is mean of two judges. Criteria-pass rate is separate. DOCX tracked-change grading fix. Distinct from AA hallucination-gated LAB. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":1463.816,"stderr":0.0,"cost_per_test":0.612386,"temperature":1.0,"top_p":1.0,"max_output_tokens":384000,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"DeepSeek","harness":null}

20.000% · raw 20 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Harvey held-out legal work protocol, internet disabled, GPT5.5 medium and Sonnet4.6 judges. Task pass requires every criterion; final is mean of two judges. Criteria-pass rate is separate. DOCX tracked-change grading fix. Distinct from AA hallucination-gated LAB. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":1129.724,"stderr":12.247,"cost_per_test":0.577367,"temperature":1.0,"top_p":1.0,"max_output_tokens":384000,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"DeepSeek","harness":null}

10.000% · raw 10 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Harvey held-out legal work protocol, internet disabled, GPT5.5 medium and Sonnet4.6 judges. Task pass requires every criterion; final is mean of two judges. Criteria-pass rate is separate. DOCX tracked-change grading fix. Distinct from AA hallucination-gated LAB. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":1407.424,"stderr":10.0,"cost_per_test":0.669758,"temperature":1.0,"top_p":1.0,"max_output_tokens":384000,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"DeepSeek","harness":null}

20.000% · raw 20 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Harvey held-out legal work protocol, internet disabled, GPT5.5 medium and Sonnet4.6 judges. Task pass requires every criterion; final is mean of two judges. Criteria-pass rate is separate. DOCX tracked-change grading fix. Distinct from AA hallucination-gated LAB. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":1214.128,"stderr":20.0,"cost_per_test":0.5812,"temperature":1.0,"top_p":1.0,"max_output_tokens":384000,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"DeepSeek","harness":null}

10.000% · raw 10 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Harvey held-out legal work protocol, internet disabled, GPT5.5 medium and Sonnet4.6 judges. Task pass requires every criterion; final is mean of two judges. Criteria-pass rate is separate. DOCX tracked-change grading fix. Distinct from AA hallucination-gated LAB. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":1437.404,"stderr":10.0,"cost_per_test":0.703428,"temperature":1.0,"top_p":1.0,"max_output_tokens":384000,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"DeepSeek","harness":null}

20.000% · raw 20 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Harvey held-out legal work protocol, internet disabled, GPT5.5 medium and Sonnet4.6 judges. Task pass requires every criterion; final is mean of two judges. Criteria-pass rate is separate. DOCX tracked-change grading fix. Distinct from AA hallucination-gated LAB. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":1358.138,"stderr":20.0,"cost_per_test":0.610376,"temperature":1.0,"top_p":1.0,"max_output_tokens":384000,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"DeepSeek","harness":null}

0.000% · raw 0 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Harvey held-out legal work protocol, internet disabled, GPT5.5 medium and Sonnet4.6 judges. Task pass requires every criterion; final is mean of two judges. Criteria-pass rate is separate. DOCX tracked-change grading fix. Distinct from AA hallucination-gated LAB. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":1431.654,"stderr":0.0,"cost_per_test":0.688859,"temperature":1.0,"top_p":1.0,"max_output_tokens":384000,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"DeepSeek","harness":null}

92.665% · raw 92.665 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Harvey held-out legal work protocol, internet disabled, GPT5.5 medium and Sonnet4.6 judges. Task pass requires every criterion; final is mean of two judges. Criteria-pass rate is separate. DOCX tracked-change grading fix. Distinct from AA hallucination-gated LAB. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":1328.108,"stderr":0.314,"cost_per_test":0.612596,"token_totals":{"input_tokens":5524481146,"output_tokens":25881985,"reasoning_tokens":10520362,"cache_read_tokens":5492827392,"cache_write_tokens":null},"temperature":1.0,"top_p":1.0,"max_output_tokens":384000,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"DeepSeek","harness":null}

93.478% · raw 93.478 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Harvey held-out legal work protocol, internet disabled, GPT5.5 medium and Sonnet4.6 judges. Task pass requires every criterion; final is mean of two judges. Criteria-pass rate is separate. DOCX tracked-change grading fix. Distinct from AA hallucination-gated LAB. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":1264.71,"stderr":1.489,"cost_per_test":0.534289,"temperature":1.0,"top_p":1.0,"max_output_tokens":384000,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"DeepSeek","harness":null}

91.756% · raw 91.756 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Harvey held-out legal work protocol, internet disabled, GPT5.5 medium and Sonnet4.6 judges. Task pass requires every criterion; final is mean of two judges. Criteria-pass rate is separate. DOCX tracked-change grading fix. Distinct from AA hallucination-gated LAB. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":1273.796,"stderr":1.65,"cost_per_test":0.66721,"temperature":1.0,"top_p":1.0,"max_output_tokens":384000,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"DeepSeek","harness":null}

74.813% · raw 74.813 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Harvey held-out legal work protocol, internet disabled, GPT5.5 medium and Sonnet4.6 judges. Task pass requires every criterion; final is mean of two judges. Criteria-pass rate is separate. DOCX tracked-change grading fix. Distinct from AA hallucination-gated LAB. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":1273.962,"stderr":2.657,"cost_per_test":0.604563,"temperature":1.0,"top_p":1.0,"max_output_tokens":384000,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"DeepSeek","harness":null}

93.426% · raw 93.426 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Harvey held-out legal work protocol, internet disabled, GPT5.5 medium and Sonnet4.6 judges. Task pass requires every criterion; final is mean of two judges. Criteria-pass rate is separate. DOCX tracked-change grading fix. Distinct from AA hallucination-gated LAB. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":1285.442,"stderr":1.46,"cost_per_test":0.640206,"temperature":1.0,"top_p":1.0,"max_output_tokens":384000,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"DeepSeek","harness":null}

94.118% · raw 94.118 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Harvey held-out legal work protocol, internet disabled, GPT5.5 medium and Sonnet4.6 judges. Task pass requires every criterion; final is mean of two judges. Criteria-pass rate is separate. DOCX tracked-change grading fix. Distinct from AA hallucination-gated LAB. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":1270.292,"stderr":1.476,"cost_per_test":0.604559,"temperature":1.0,"top_p":1.0,"max_output_tokens":384000,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"DeepSeek","harness":null}

90.582% · raw 90.582 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Harvey held-out legal work protocol, internet disabled, GPT5.5 medium and Sonnet4.6 judges. Task pass requires every criterion; final is mean of two judges. Criteria-pass rate is separate. DOCX tracked-change grading fix. Distinct from AA hallucination-gated LAB. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":1497.44,"stderr":1.712,"cost_per_test":0.70205,"temperature":1.0,"top_p":1.0,"max_output_tokens":384000,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"DeepSeek","harness":null}

95.338% · raw 95.338 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Harvey held-out legal work protocol, internet disabled, GPT5.5 medium and Sonnet4.6 judges. Task pass requires every criterion; final is mean of two judges. Criteria-pass rate is separate. DOCX tracked-change grading fix. Distinct from AA hallucination-gated LAB. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":1348.014,"stderr":1.197,"cost_per_test":0.64235,"temperature":1.0,"top_p":1.0,"max_output_tokens":384000,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"DeepSeek","harness":null}

97.527% · raw 97.527 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Harvey held-out legal work protocol, internet disabled, GPT5.5 medium and Sonnet4.6 judges. Task pass requires every criterion; final is mean of two judges. Criteria-pass rate is separate. DOCX tracked-change grading fix. Distinct from AA hallucination-gated LAB. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":1228.252,"stderr":0.925,"cost_per_test":0.52666,"temperature":1.0,"top_p":1.0,"max_output_tokens":384000,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"DeepSeek","harness":null}

94.881% · raw 94.881 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Harvey held-out legal work protocol, internet disabled, GPT5.5 medium and Sonnet4.6 judges. Task pass requires every criterion; final is mean of two judges. Criteria-pass rate is separate. DOCX tracked-change grading fix. Distinct from AA hallucination-gated LAB. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":1539.008,"stderr":1.29,"cost_per_test":0.633521,"temperature":1.0,"top_p":1.0,"max_output_tokens":384000,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"DeepSeek","harness":null}

90.637% · raw 90.637 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Harvey held-out legal work protocol, internet disabled, GPT5.5 medium and Sonnet4.6 judges. Task pass requires every criterion; final is mean of two judges. Criteria-pass rate is separate. DOCX tracked-change grading fix. Distinct from AA hallucination-gated LAB. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":1291.326,"stderr":1.786,"cost_per_test":0.520216,"temperature":1.0,"top_p":1.0,"max_output_tokens":384000,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"DeepSeek","harness":null}

98.020% · raw 98.02 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Harvey held-out legal work protocol, internet disabled, GPT5.5 medium and Sonnet4.6 judges. Task pass requires every criterion; final is mean of two judges. Criteria-pass rate is separate. DOCX tracked-change grading fix. Distinct from AA hallucination-gated LAB. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":1272.778,"stderr":0.802,"cost_per_test":0.69079,"temperature":1.0,"top_p":1.0,"max_output_tokens":384000,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"DeepSeek","harness":null}

93.011% · raw 93.011 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Harvey held-out legal work protocol, internet disabled, GPT5.5 medium and Sonnet4.6 judges. Task pass requires every criterion; final is mean of two judges. Criteria-pass rate is separate. DOCX tracked-change grading fix. Distinct from AA hallucination-gated LAB. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":1433.79,"stderr":1.529,"cost_per_test":0.635298,"temperature":1.0,"top_p":1.0,"max_output_tokens":384000,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"DeepSeek","harness":null}

89.796% · raw 89.796 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Harvey held-out legal work protocol, internet disabled, GPT5.5 medium and Sonnet4.6 judges. Task pass requires every criterion; final is mean of two judges. Criteria-pass rate is separate. DOCX tracked-change grading fix. Distinct from AA hallucination-gated LAB. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":1356.48,"stderr":1.637,"cost_per_test":0.654524,"temperature":1.0,"top_p":1.0,"max_output_tokens":384000,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"DeepSeek","harness":null}

95.973% · raw 95.973 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Harvey held-out legal work protocol, internet disabled, GPT5.5 medium and Sonnet4.6 judges. Task pass requires every criterion; final is mean of two judges. Criteria-pass rate is separate. DOCX tracked-change grading fix. Distinct from AA hallucination-gated LAB. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":1457.208,"stderr":1.141,"cost_per_test":0.656838,"temperature":1.0,"top_p":1.0,"max_output_tokens":384000,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"DeepSeek","harness":null}

92.545% · raw 92.545 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Harvey held-out legal work protocol, internet disabled, GPT5.5 medium and Sonnet4.6 judges. Task pass requires every criterion; final is mean of two judges. Criteria-pass rate is separate. DOCX tracked-change grading fix. Distinct from AA hallucination-gated LAB. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":1218.928,"stderr":1.587,"cost_per_test":0.468313,"temperature":1.0,"top_p":1.0,"max_output_tokens":384000,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"DeepSeek","harness":null}

94.485% · raw 94.485 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Harvey held-out legal work protocol, internet disabled, GPT5.5 medium and Sonnet4.6 judges. Task pass requires every criterion; final is mean of two judges. Criteria-pass rate is separate. DOCX tracked-change grading fix. Distinct from AA hallucination-gated LAB. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":1109.874,"stderr":1.387,"cost_per_test":0.476309,"temperature":1.0,"top_p":1.0,"max_output_tokens":384000,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"DeepSeek","harness":null}

94.840% · raw 94.84 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Harvey held-out legal work protocol, internet disabled, GPT5.5 medium and Sonnet4.6 judges. Task pass requires every criterion; final is mean of two judges. Criteria-pass rate is separate. DOCX tracked-change grading fix. Distinct from AA hallucination-gated LAB. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":1311.01,"stderr":1.322,"cost_per_test":0.601227,"temperature":1.0,"top_p":1.0,"max_output_tokens":384000,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"DeepSeek","harness":null}

94.821% · raw 94.821 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Harvey held-out legal work protocol, internet disabled, GPT5.5 medium and Sonnet4.6 judges. Task pass requires every criterion; final is mean of two judges. Criteria-pass rate is separate. DOCX tracked-change grading fix. Distinct from AA hallucination-gated LAB. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":1463.816,"stderr":1.402,"cost_per_test":0.612386,"temperature":1.0,"top_p":1.0,"max_output_tokens":384000,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"DeepSeek","harness":null}

92.481% · raw 92.481 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Harvey held-out legal work protocol, internet disabled, GPT5.5 medium and Sonnet4.6 judges. Task pass requires every criterion; final is mean of two judges. Criteria-pass rate is separate. DOCX tracked-change grading fix. Distinct from AA hallucination-gated LAB. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":1129.724,"stderr":1.62,"cost_per_test":0.577367,"temperature":1.0,"top_p":1.0,"max_output_tokens":384000,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"DeepSeek","harness":null}

94.752% · raw 94.752 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Harvey held-out legal work protocol, internet disabled, GPT5.5 medium and Sonnet4.6 judges. Task pass requires every criterion; final is mean of two judges. Criteria-pass rate is separate. DOCX tracked-change grading fix. Distinct from AA hallucination-gated LAB. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":1407.424,"stderr":1.206,"cost_per_test":0.669758,"temperature":1.0,"top_p":1.0,"max_output_tokens":384000,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"DeepSeek","harness":null}

92.903% · raw 92.903 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Harvey held-out legal work protocol, internet disabled, GPT5.5 medium and Sonnet4.6 judges. Task pass requires every criterion; final is mean of two judges. Criteria-pass rate is separate. DOCX tracked-change grading fix. Distinct from AA hallucination-gated LAB. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":1214.128,"stderr":1.461,"cost_per_test":0.5812,"temperature":1.0,"top_p":1.0,"max_output_tokens":384000,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"DeepSeek","harness":null}

87.793% · raw 87.793 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Harvey held-out legal work protocol, internet disabled, GPT5.5 medium and Sonnet4.6 judges. Task pass requires every criterion; final is mean of two judges. Criteria-pass rate is separate. DOCX tracked-change grading fix. Distinct from AA hallucination-gated LAB. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":1437.404,"stderr":1.896,"cost_per_test":0.703428,"temperature":1.0,"top_p":1.0,"max_output_tokens":384000,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"DeepSeek","harness":null}

95.420% · raw 95.42 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Harvey held-out legal work protocol, internet disabled, GPT5.5 medium and Sonnet4.6 judges. Task pass requires every criterion; final is mean of two judges. Criteria-pass rate is separate. DOCX tracked-change grading fix. Distinct from AA hallucination-gated LAB. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":1358.138,"stderr":1.294,"cost_per_test":0.610376,"temperature":1.0,"top_p":1.0,"max_output_tokens":384000,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"DeepSeek","harness":null}

89.899% · raw 89.899 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Harvey held-out legal work protocol, internet disabled, GPT5.5 medium and Sonnet4.6 judges. Task pass requires every criterion; final is mean of two judges. Criteria-pass rate is separate. DOCX tracked-change grading fix. Distinct from AA hallucination-gated LAB. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":1431.654,"stderr":1.752,"cost_per_test":0.688859,"temperature":1.0,"top_p":1.0,"max_output_tokens":384000,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"DeepSeek","harness":null}

44.350% · raw 44.35 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Private193-item tax suite. All-pass accuracy and must-pass weighted partial credit are separate; both multiply by citation factor0.7+0.3*citation quality. No inferred tax professional reliability. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":1264.434,"stderr":3.574,"cost_per_test":0.459454,"token_totals":{"input_tokens":779160448,"output_tokens":53561725,"reasoning_tokens":45716077,"cache_read_tokens":712066688,"cache_write_tokens":null},"temperature":1.0,"top_p":1.0,"max_output_tokens":384000,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"DeepSeek","harness":null}

$0.459454/task · raw 0.459454 USD/task

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Private193-item tax suite. All-pass accuracy and must-pass weighted partial credit are separate; both multiply by citation factor0.7+0.3*citation quality. No inferred tax professional reliability. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":1264.434,"stderr":3.574,"cost_per_test":0.459454,"token_totals":{"input_tokens":779160448,"output_tokens":53561725,"reasoning_tokens":45716077,"cache_read_tokens":712066688,"cache_write_tokens":null},"temperature":1.0,"top_p":1.0,"max_output_tokens":384000,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"DeepSeek","harness":null} Source score standard error describes the benchmark quality score, not uncertainty in cost or latency.

1264.434 s · raw 1264.434 s

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Private193-item tax suite. All-pass accuracy and must-pass weighted partial credit are separate; both multiply by citation factor0.7+0.3*citation quality. No inferred tax professional reliability. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":1264.434,"stderr":3.574,"cost_per_test":0.459454,"token_totals":{"input_tokens":779160448,"output_tokens":53561725,"reasoning_tokens":45716077,"cache_read_tokens":712066688,"cache_write_tokens":null},"temperature":1.0,"top_p":1.0,"max_output_tokens":384000,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"DeepSeek","harness":null} Source score standard error describes the benchmark quality score, not uncertainty in cost or latency.

55.767% · raw 55.767 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Private193-item tax suite. All-pass accuracy and must-pass weighted partial credit are separate; both multiply by citation factor0.7+0.3*citation quality. No inferred tax professional reliability. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":1324.466,"stderr":7.657,"cost_per_test":0.460654,"temperature":1.0,"top_p":1.0,"max_output_tokens":384000,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"DeepSeek","harness":null}

46.667% · raw 46.667 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Private193-item tax suite. All-pass accuracy and must-pass weighted partial credit are separate; both multiply by citation factor0.7+0.3*citation quality. No inferred tax professional reliability. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":1361.742,"stderr":9.264,"cost_per_test":0.492773,"temperature":1.0,"top_p":1.0,"max_output_tokens":384000,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"DeepSeek","harness":null}

56.386% · raw 56.386 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Private193-item tax suite. All-pass accuracy and must-pass weighted partial credit are separate; both multiply by citation factor0.7+0.3*citation quality. No inferred tax professional reliability. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":1540.94,"stderr":9.158,"cost_per_test":0.541685,"temperature":1.0,"top_p":1.0,"max_output_tokens":384000,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"DeepSeek","harness":null}

45.667% · raw 45.667 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Private193-item tax suite. All-pass accuracy and must-pass weighted partial credit are separate; both multiply by citation factor0.7+0.3*citation quality. No inferred tax professional reliability. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":677.794,"stderr":9.118,"cost_per_test":0.241521,"temperature":1.0,"top_p":1.0,"max_output_tokens":384000,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"DeepSeek","harness":null}

33.333% · raw 33.333 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Private193-item tax suite. All-pass accuracy and must-pass weighted partial credit are separate; both multiply by citation factor0.7+0.3*citation quality. No inferred tax professional reliability. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":1400.675,"stderr":8.754,"cost_per_test":0.547143,"temperature":1.0,"top_p":1.0,"max_output_tokens":384000,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"DeepSeek","harness":null}

23.333% · raw 23.333 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Private193-item tax suite. All-pass accuracy and must-pass weighted partial credit are separate; both multiply by citation factor0.7+0.3*citation quality. No inferred tax professional reliability. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":1323.029,"stderr":7.854,"cost_per_test":0.472427,"temperature":1.0,"top_p":1.0,"max_output_tokens":384000,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"DeepSeek","harness":null}

74.908% · raw 74.908 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Private193-item tax suite. All-pass accuracy and must-pass weighted partial credit are separate; both multiply by citation factor0.7+0.3*citation quality. No inferred tax professional reliability. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":1264.434,"stderr":2.917,"cost_per_test":0.459454,"token_totals":{"input_tokens":779160448,"output_tokens":53561725,"reasoning_tokens":45716077,"cache_read_tokens":712066688,"cache_write_tokens":null},"temperature":1.0,"top_p":1.0,"max_output_tokens":384000,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"DeepSeek","harness":null}

72.524% · raw 72.524 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Private193-item tax suite. All-pass accuracy and must-pass weighted partial credit are separate; both multiply by citation factor0.7+0.3*citation quality. No inferred tax professional reliability. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":1324.466,"stderr":6.621,"cost_per_test":0.460654,"temperature":1.0,"top_p":1.0,"max_output_tokens":384000,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"DeepSeek","harness":null}

70.819% · raw 70.819 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Private193-item tax suite. All-pass accuracy and must-pass weighted partial credit are separate; both multiply by citation factor0.7+0.3*citation quality. No inferred tax professional reliability. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":1361.742,"stderr":7.976,"cost_per_test":0.492773,"temperature":1.0,"top_p":1.0,"max_output_tokens":384000,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"DeepSeek","harness":null}

75.003% · raw 75.003 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Private193-item tax suite. All-pass accuracy and must-pass weighted partial credit are separate; both multiply by citation factor0.7+0.3*citation quality. No inferred tax professional reliability. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":1540.94,"stderr":7.702,"cost_per_test":0.541685,"temperature":1.0,"top_p":1.0,"max_output_tokens":384000,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"DeepSeek","harness":null}

79.503% · raw 79.503 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Private193-item tax suite. All-pass accuracy and must-pass weighted partial credit are separate; both multiply by citation factor0.7+0.3*citation quality. No inferred tax professional reliability. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":677.794,"stderr":6.7,"cost_per_test":0.241521,"temperature":1.0,"top_p":1.0,"max_output_tokens":384000,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"DeepSeek","harness":null}

79.399% · raw 79.399 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Private193-item tax suite. All-pass accuracy and must-pass weighted partial credit are separate; both multiply by citation factor0.7+0.3*citation quality. No inferred tax professional reliability. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":1400.675,"stderr":6.669,"cost_per_test":0.547143,"temperature":1.0,"top_p":1.0,"max_output_tokens":384000,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"DeepSeek","harness":null}

73.236% · raw 73.236 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Private193-item tax suite. All-pass accuracy and must-pass weighted partial credit are separate; both multiply by citation factor0.7+0.3*citation quality. No inferred tax professional reliability. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":1323.029,"stderr":7.536,"cost_per_test":0.472427,"temperature":1.0,"top_p":1.0,"max_output_tokens":384000,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"DeepSeek","harness":null}

87.730% · raw 87.73 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: OpenHands. Independent model-plus-harness web-application evaluation. Preserve OpenHands versus other agent variants; source costs/runtime are task observations, not API tariffs. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":973.834,"stderr":2.752,"cost_per_test":0.579984,"token_totals":{"input_tokens":2173326299,"output_tokens":9246207,"reasoning_tokens":3107112,"cache_read_tokens":2156782720,"cache_write_tokens":null},"temperature":1.0,"top_p":1.0,"max_output_tokens":384000,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"DeepSeek","harness":"OpenHands"}

$0.579984/task · raw 0.579984 USD/task

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: OpenHands. Independent model-plus-harness web-application evaluation. Preserve OpenHands versus other agent variants; source costs/runtime are task observations, not API tariffs. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":973.834,"stderr":2.752,"cost_per_test":0.579984,"token_totals":{"input_tokens":2173326299,"output_tokens":9246207,"reasoning_tokens":3107112,"cache_read_tokens":2156782720,"cache_write_tokens":null},"temperature":1.0,"top_p":1.0,"max_output_tokens":384000,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"DeepSeek","harness":"OpenHands"} Source score standard error describes the benchmark quality score, not uncertainty in cost or latency.

973.834 s · raw 973.834 s

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: OpenHands. Independent model-plus-harness web-application evaluation. Preserve OpenHands versus other agent variants; source costs/runtime are task observations, not API tariffs. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":973.834,"stderr":2.752,"cost_per_test":0.579984,"token_totals":{"input_tokens":2173326299,"output_tokens":9246207,"reasoning_tokens":3107112,"cache_read_tokens":2156782720,"cache_write_tokens":null},"temperature":1.0,"top_p":1.0,"max_output_tokens":384000,"reasoning":null,"reasoning_effort":"max","verbosity":null,"compute_effort":null,"provider":"DeepSeek","harness":"OpenHands"} Source score standard error describes the benchmark quality score, not uncertainty in cost or latency.

3.89 acronyms/response · raw 3.89 acronyms/response

Headline · max effort · Independent evaluator

Source/record date: 2026-10-19

Public rendered chart labels manually captured by the root agent, then transcribed once from those captures. Published 19 October 2026; source checked 20 October 2026; execution dates unpublished. No system prompt: one response to each of 900 English questions, 30 questions per each of 30 traits. This describes default communication style and trait prominence, not objective answer quality or accuracy. No AA API, internal app-state access, automated collector, paid export or inference was used. Programmatically measured Acronyms: Two or more capital letters in a row (AI, USA) or dotted forms (U.S.). Source-native average over responses to the same900questions; heatmap colors use pool-relative percentiles, but the visible cell numbers here are native quantities, not those color percentiles. The Characters definition explicitly removes code and URLs; exact feature-specific handling follows the methodology. Average sentence length is words per sentence, not per response. Ends with a question is the share of responses (%); bold/italic count spans, not words or characters. Descriptive communication style, not a quality ranking. Original Flash and Flash1017 are distinct explicitly named checkpoints and retain separate IDs. Source label: 3.89.

13.80 words/sentence · raw 13.8 words/sentence

Headline · max effort · Independent evaluator

Source/record date: 2026-10-19

Public rendered chart labels manually captured by the root agent, then transcribed once from those captures. Published 19 October 2026; source checked 20 October 2026; execution dates unpublished. No system prompt: one response to each of 900 English questions, 30 questions per each of 30 traits. This describes default communication style and trait prominence, not objective answer quality or accuracy. No AA API, internal app-state access, automated collector, paid export or inference was used. Programmatically measured Avg. sentence length: Words divided by sentences. Source-native average over responses to the same900questions; heatmap colors use pool-relative percentiles, but the visible cell numbers here are native quantities, not those color percentiles. The Characters definition explicitly removes code and URLs; exact feature-specific handling follows the methodology. Average sentence length is words per sentence, not per response. Ends with a question is the share of responses (%); bold/italic count spans, not words or characters. Descriptive communication style, not a quality ranking. Original Flash and Flash1017 are distinct explicitly named checkpoints and retain separate IDs. Source label: 13.8.

12.70 bold spans/response · raw 12.7 bold spans/response

Headline · max effort · Independent evaluator

Source/record date: 2026-10-19

Public rendered chart labels manually captured by the root agent, then transcribed once from those captures. Published 19 October 2026; source checked 20 October 2026; execution dates unpublished. No system prompt: one response to each of 900 English questions, 30 questions per each of 30 traits. This describes default communication style and trait prominence, not objective answer quality or accuracy. No AA API, internal app-state access, automated collector, paid export or inference was used. Programmatically measured Bold text: Uses of bold: each unbroken run of bold text counts once, so a single bolded word and a fully bolded sentence both count as 1. Source-native average over responses to the same900questions; heatmap colors use pool-relative percentiles, but the visible cell numbers here are native quantities, not those color percentiles. The Characters definition explicitly removes code and URLs; exact feature-specific handling follows the methodology. Average sentence length is words per sentence, not per response. Ends with a question is the share of responses (%); bold/italic count spans, not words or characters. Descriptive communication style, not a quality ranking. Original Flash and Flash1017 are distinct explicitly named checkpoints and retain separate IDs. Source label: 12.7.

3639 characters/response · raw 3639 characters/response

Headline · max effort · Independent evaluator

Source/record date: 2026-10-19

Public rendered chart labels manually captured by the root agent, then transcribed once from those captures. Published 19 October 2026; source checked 20 October 2026; execution dates unpublished. No system prompt: one response to each of 900 English questions, 30 questions per each of 30 traits. This describes default communication style and trait prominence, not objective answer quality or accuracy. No AA API, internal app-state access, automated collector, paid export or inference was used. Programmatically measured Characters: Characters in the prose, spaces included, after code and URLs are removed. Source-native average over responses to the same900questions; heatmap colors use pool-relative percentiles, but the visible cell numbers here are native quantities, not those color percentiles. The Characters definition explicitly removes code and URLs; exact feature-specific handling follows the methodology. Average sentence length is words per sentence, not per response. Ends with a question is the share of responses (%); bold/italic count spans, not words or characters. Descriptive communication style, not a quality ranking. Original Flash and Flash1017 are distinct explicitly named checkpoints and retain separate IDs. Source label: 3,639.

9.72 colons/response · raw 9.72 colons/response

Headline · max effort · Independent evaluator

Source/record date: 2026-10-19

Public rendered chart labels manually captured by the root agent, then transcribed once from those captures. Published 19 October 2026; source checked 20 October 2026; execution dates unpublished. No system prompt: one response to each of 900 English questions, 30 questions per each of 30 traits. This describes default communication style and trait prominence, not objective answer quality or accuracy. No AA API, internal app-state access, automated collector, paid export or inference was used. Programmatically measured Colons: Colons. Source-native average over responses to the same900questions; heatmap colors use pool-relative percentiles, but the visible cell numbers here are native quantities, not those color percentiles. The Characters definition explicitly removes code and URLs; exact feature-specific handling follows the methodology. Average sentence length is words per sentence, not per response. Ends with a question is the share of responses (%); bold/italic count spans, not words or characters. Descriptive communication style, not a quality ranking. Original Flash and Flash1017 are distinct explicitly named checkpoints and retain separate IDs. Source label: 9.72.

8.92 contractions/response · raw 8.92 contractions/response

Headline · max effort · Independent evaluator

Source/record date: 2026-10-19

Public rendered chart labels manually captured by the root agent, then transcribed once from those captures. Published 19 October 2026; source checked 20 October 2026; execution dates unpublished. No system prompt: one response to each of 900 English questions, 30 questions per each of 30 traits. This describes default communication style and trait prominence, not objective answer quality or accuracy. No AA API, internal app-state access, automated collector, paid export or inference was used. Programmatically measured Contractions: Contractions such as “don’t”, “you’re” and “it’s”; possessives such as “John’s” do not count. Source-native average over responses to the same900questions; heatmap colors use pool-relative percentiles, but the visible cell numbers here are native quantities, not those color percentiles. The Characters definition explicitly removes code and URLs; exact feature-specific handling follows the methodology. Average sentence length is words per sentence, not per response. Ends with a question is the share of responses (%); bold/italic count spans, not words or characters. Descriptive communication style, not a quality ranking. Original Flash and Flash1017 are distinct explicitly named checkpoints and retain separate IDs. Source label: 8.92.

0.10 ellipses/response · raw 0.1 ellipses/response

Headline · max effort · Independent evaluator

Source/record date: 2026-10-19

Public rendered chart labels manually captured by the root agent, then transcribed once from those captures. Published 19 October 2026; source checked 20 October 2026; execution dates unpublished. No system prompt: one response to each of 900 English questions, 30 questions per each of 30 traits. This describes default communication style and trait prominence, not objective answer quality or accuracy. No AA API, internal app-state access, automated collector, paid export or inference was used. Programmatically measured Ellipses: Three periods (...) or the ellipsis character (…). Source-native average over responses to the same900questions; heatmap colors use pool-relative percentiles, but the visible cell numbers here are native quantities, not those color percentiles. The Characters definition explicitly removes code and URLs; exact feature-specific handling follows the methodology. Average sentence length is words per sentence, not per response. Ends with a question is the share of responses (%); bold/italic count spans, not words or characters. Descriptive communication style, not a quality ranking. Original Flash and Flash1017 are distinct explicitly named checkpoints and retain separate IDs. Source label: 0.10.

5.74 em dashes/response · raw 5.74 em dashes/response

Headline · max effort · Independent evaluator

Source/record date: 2026-10-19

Public rendered chart labels manually captured by the root agent, then transcribed once from those captures. Published 19 October 2026; source checked 20 October 2026; execution dates unpublished. No system prompt: one response to each of 900 English questions, 30 questions per each of 30 traits. This describes default communication style and trait prominence, not objective answer quality or accuracy. No AA API, internal app-state access, automated collector, paid export or inference was used. Programmatically measured Em dashes: The em dash character (—); double hyphens do not count. Source-native average over responses to the same900questions; heatmap colors use pool-relative percentiles, but the visible cell numbers here are native quantities, not those color percentiles. The Characters definition explicitly removes code and URLs; exact feature-specific handling follows the methodology. Average sentence length is words per sentence, not per response. Ends with a question is the share of responses (%); bold/italic count spans, not words or characters. Descriptive communication style, not a quality ranking. Original Flash and Flash1017 are distinct explicitly named checkpoints and retain separate IDs. Source label: 5.74.

0.01 emojis/response · raw 0.01 emojis/response

Headline · max effort · Independent evaluator

Source/record date: 2026-10-19

Public rendered chart labels manually captured by the root agent, then transcribed once from those captures. Published 19 October 2026; source checked 20 October 2026; execution dates unpublished. No system prompt: one response to each of 900 English questions, 30 questions per each of 30 traits. This describes default communication style and trait prominence, not objective answer quality or accuracy. No AA API, internal app-state access, automated collector, paid export or inference was used. Programmatically measured Emojis: Each emoji as the reader sees it. An emoji built from several characters, such as a family (👨‍👩‍👧), still counts as one emoji. Source-native average over responses to the same900questions; heatmap colors use pool-relative percentiles, but the visible cell numbers here are native quantities, not those color percentiles. The Characters definition explicitly removes code and URLs; exact feature-specific handling follows the methodology. Average sentence length is words per sentence, not per response. Ends with a question is the share of responses (%); bold/italic count spans, not words or characters. Descriptive communication style, not a quality ranking. Original Flash and Flash1017 are distinct explicitly named checkpoints and retain separate IDs. Source label: 0.01.

0% · raw 0 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-19

Public rendered chart labels manually captured by the root agent, then transcribed once from those captures. Published 19 October 2026; source checked 20 October 2026; execution dates unpublished. No system prompt: one response to each of 900 English questions, 30 questions per each of 30 traits. This describes default communication style and trait prominence, not objective answer quality or accuracy. No AA API, internal app-state access, automated collector, paid export or inference was used. Programmatically measured Ends with a question: Whether the answer’s last character is a question mark, reported as the share of responses that do. Source-native average over responses to the same900questions; heatmap colors use pool-relative percentiles, but the visible cell numbers here are native quantities, not those color percentiles. The Characters definition explicitly removes code and URLs; exact feature-specific handling follows the methodology. Average sentence length is words per sentence, not per response. Ends with a question is the share of responses (%); bold/italic count spans, not words or characters. Descriptive communication style, not a quality ranking. Original Flash and Flash1017 are distinct explicitly named checkpoints and retain separate IDs. Source label: 0%.

0.04 exclamations/response · raw 0.04 exclamations/response

Headline · max effort · Independent evaluator

Source/record date: 2026-10-19

Public rendered chart labels manually captured by the root agent, then transcribed once from those captures. Published 19 October 2026; source checked 20 October 2026; execution dates unpublished. No system prompt: one response to each of 900 English questions, 30 questions per each of 30 traits. This describes default communication style and trait prominence, not objective answer quality or accuracy. No AA API, internal app-state access, automated collector, paid export or inference was used. Programmatically measured Exclamations: Exclamation marks. Source-native average over responses to the same900questions; heatmap colors use pool-relative percentiles, but the visible cell numbers here are native quantities, not those color percentiles. The Characters definition explicitly removes code and URLs; exact feature-specific handling follows the methodology. Average sentence length is words per sentence, not per response. Ends with a question is the share of responses (%); bold/italic count spans, not words or characters. Descriptive communication style, not a quality ranking. Original Flash and Flash1017 are distinct explicitly named checkpoints and retain separate IDs. Source label: 0.04.

2.47 headings/response · raw 2.47 headings/response

Headline · max effort · Independent evaluator

Source/record date: 2026-10-19

Public rendered chart labels manually captured by the root agent, then transcribed once from those captures. Published 19 October 2026; source checked 20 October 2026; execution dates unpublished. No system prompt: one response to each of 900 English questions, 30 questions per each of 30 traits. This describes default communication style and trait prominence, not objective answer quality or accuracy. No AA API, internal app-state access, automated collector, paid export or inference was used. Programmatically measured Headings: Headings of any level. Source-native average over responses to the same900questions; heatmap colors use pool-relative percentiles, but the visible cell numbers here are native quantities, not those color percentiles. The Characters definition explicitly removes code and URLs; exact feature-specific handling follows the methodology. Average sentence length is words per sentence, not per response. Ends with a question is the share of responses (%); bold/italic count spans, not words or characters. Descriptive communication style, not a quality ranking. Original Flash and Flash1017 are distinct explicitly named checkpoints and retain separate IDs. Source label: 2.47.

0.72 italic spans/response · raw 0.72 italic spans/response

Headline · max effort · Independent evaluator

Source/record date: 2026-10-19

Public rendered chart labels manually captured by the root agent, then transcribed once from those captures. Published 19 October 2026; source checked 20 October 2026; execution dates unpublished. No system prompt: one response to each of 900 English questions, 30 questions per each of 30 traits. This describes default communication style and trait prominence, not objective answer quality or accuracy. No AA API, internal app-state access, automated collector, paid export or inference was used. Programmatically measured Italics: Uses of italics: each unbroken run of italic text counts once, so a single italicized word and a fully italicized sentence both count as 1. Source-native average over responses to the same900questions; heatmap colors use pool-relative percentiles, but the visible cell numbers here are native quantities, not those color percentiles. The Characters definition explicitly removes code and URLs; exact feature-specific handling follows the methodology. Average sentence length is words per sentence, not per response. Ends with a question is the share of responses (%); bold/italic count spans, not words or characters. Descriptive communication style, not a quality ranking. Original Flash and Flash1017 are distinct explicitly named checkpoints and retain separate IDs. Source label: 0.72.

2.65 lists/response · raw 2.65 lists/response

Headline · max effort · Independent evaluator

Source/record date: 2026-10-19

Public rendered chart labels manually captured by the root agent, then transcribed once from those captures. Published 19 October 2026; source checked 20 October 2026; execution dates unpublished. No system prompt: one response to each of 900 English questions, 30 questions per each of 30 traits. This describes default communication style and trait prominence, not objective answer quality or accuracy. No AA API, internal app-state access, automated collector, paid export or inference was used. Programmatically measured Lists: Top-level bulleted or numbered lists; nested sub-lists do not count separately. Source-native average over responses to the same900questions; heatmap colors use pool-relative percentiles, but the visible cell numbers here are native quantities, not those color percentiles. The Characters definition explicitly removes code and URLs; exact feature-specific handling follows the methodology. Average sentence length is words per sentence, not per response. Ends with a question is the share of responses (%); bold/italic count spans, not words or characters. Descriptive communication style, not a quality ranking. Original Flash and Flash1017 are distinct explicitly named checkpoints and retain separate IDs. Source label: 2.65.

9.58 paragraphs/response · raw 9.58 paragraphs/response

Headline · max effort · Independent evaluator

Source/record date: 2026-10-19

Public rendered chart labels manually captured by the root agent, then transcribed once from those captures. Published 19 October 2026; source checked 20 October 2026; execution dates unpublished. No system prompt: one response to each of 900 English questions, 30 questions per each of 30 traits. This describes default communication style and trait prominence, not objective answer quality or accuracy. No AA API, internal app-state access, automated collector, paid export or inference was used. Programmatically measured Paragraphs: Paragraphs of prose; headings, code and the items of a single-spaced list do not count. Source-native average over responses to the same900questions; heatmap colors use pool-relative percentiles, but the visible cell numbers here are native quantities, not those color percentiles. The Characters definition explicitly removes code and URLs; exact feature-specific handling follows the methodology. Average sentence length is words per sentence, not per response. Ends with a question is the share of responses (%); bold/italic count spans, not words or characters. Descriptive communication style, not a quality ranking. Original Flash and Flash1017 are distinct explicitly named checkpoints and retain separate IDs. Source label: 9.58.

0.79 parenthesis pairs/response · raw 0.79 parenthesis pairs/response

Headline · max effort · Independent evaluator

Source/record date: 2026-10-19

Public rendered chart labels manually captured by the root agent, then transcribed once from those captures. Published 19 October 2026; source checked 20 October 2026; execution dates unpublished. No system prompt: one response to each of 900 English questions, 30 questions per each of 30 traits. This describes default communication style and trait prominence, not objective answer quality or accuracy. No AA API, internal app-state access, automated collector, paid export or inference was used. Programmatically measured Parentheses: Closed pairs of parentheses, nested pairs included; unmatched parentheses do not count. Source-native average over responses to the same900questions; heatmap colors use pool-relative percentiles, but the visible cell numbers here are native quantities, not those color percentiles. The Characters definition explicitly removes code and URLs; exact feature-specific handling follows the methodology. Average sentence length is words per sentence, not per response. Ends with a question is the share of responses (%); bold/italic count spans, not words or characters. Descriptive communication style, not a quality ranking. Original Flash and Flash1017 are distinct explicitly named checkpoints and retain separate IDs. Source label: 0.79.

3.20 question marks/response · raw 3.2 question marks/response

Headline · max effort · Independent evaluator

Source/record date: 2026-10-19

Public rendered chart labels manually captured by the root agent, then transcribed once from those captures. Published 19 October 2026; source checked 20 October 2026; execution dates unpublished. No system prompt: one response to each of 900 English questions, 30 questions per each of 30 traits. This describes default communication style and trait prominence, not objective answer quality or accuracy. No AA API, internal app-state access, automated collector, paid export or inference was used. Programmatically measured Questions: Question marks. Source-native average over responses to the same900questions; heatmap colors use pool-relative percentiles, but the visible cell numbers here are native quantities, not those color percentiles. The Characters definition explicitly removes code and URLs; exact feature-specific handling follows the methodology. Average sentence length is words per sentence, not per response. Ends with a question is the share of responses (%); bold/italic count spans, not words or characters. Descriptive communication style, not a quality ranking. Original Flash and Flash1017 are distinct explicitly named checkpoints and retain separate IDs. Source label: 3.20.

13.40 quotation marks/response · raw 13.4 quotation marks/response

Headline · max effort · Independent evaluator

Source/record date: 2026-10-19

Public rendered chart labels manually captured by the root agent, then transcribed once from those captures. Published 19 October 2026; source checked 20 October 2026; execution dates unpublished. No system prompt: one response to each of 900 English questions, 30 questions per each of 30 traits. This describes default communication style and trait prominence, not objective answer quality or accuracy. No AA API, internal app-state access, automated collector, paid export or inference was used. Programmatically measured Quotes: Quotation marks of any style, such as straight, curly, « » and „ “; apostrophes do not count. Source-native average over responses to the same900questions; heatmap colors use pool-relative percentiles, but the visible cell numbers here are native quantities, not those color percentiles. The Characters definition explicitly removes code and URLs; exact feature-specific handling follows the methodology. Average sentence length is words per sentence, not per response. Ends with a question is the share of responses (%); bold/italic count spans, not words or characters. Descriptive communication style, not a quality ranking. Original Flash and Flash1017 are distinct explicitly named checkpoints and retain separate IDs. Source label: 13.4.

41.90 sentences/response · raw 41.9 sentences/response

Headline · max effort · Independent evaluator

Source/record date: 2026-10-19

Public rendered chart labels manually captured by the root agent, then transcribed once from those captures. Published 19 October 2026; source checked 20 October 2026; execution dates unpublished. No system prompt: one response to each of 900 English questions, 30 questions per each of 30 traits. This describes default communication style and trait prominence, not objective answer quality or accuracy. No AA API, internal app-state access, automated collector, paid export or inference was used. Programmatically measured Sentences: Sentences found by a rule-based splitter that handles abbreviations such as “Dr.”. Source-native average over responses to the same900questions; heatmap colors use pool-relative percentiles, but the visible cell numbers here are native quantities, not those color percentiles. The Characters definition explicitly removes code and URLs; exact feature-specific handling follows the methodology. Average sentence length is words per sentence, not per response. Ends with a question is the share of responses (%); bold/italic count spans, not words or characters. Descriptive communication style, not a quality ranking. Original Flash and Flash1017 are distinct explicitly named checkpoints and retain separate IDs. Source label: 41.9.

0.12 tables/response · raw 0.12 tables/response

Headline · max effort · Independent evaluator

Source/record date: 2026-10-19

Public rendered chart labels manually captured by the root agent, then transcribed once from those captures. Published 19 October 2026; source checked 20 October 2026; execution dates unpublished. No system prompt: one response to each of 900 English questions, 30 questions per each of 30 traits. This describes default communication style and trait prominence, not objective answer quality or accuracy. No AA API, internal app-state access, automated collector, paid export or inference was used. Programmatically measured Tables: Tables. Source-native average over responses to the same900questions; heatmap colors use pool-relative percentiles, but the visible cell numbers here are native quantities, not those color percentiles. The Characters definition explicitly removes code and URLs; exact feature-specific handling follows the methodology. Average sentence length is words per sentence, not per response. Ends with a question is the share of responses (%); bold/italic count spans, not words or characters. Descriptive communication style, not a quality ranking. Original Flash and Flash1017 are distinct explicitly named checkpoints and retain separate IDs. Source label: 0.12.

537 words/response · raw 537 words/response

Headline · max effort · Independent evaluator

Source/record date: 2026-10-19

Public rendered chart labels manually captured by the root agent, then transcribed once from those captures. Published 19 October 2026; source checked 20 October 2026; execution dates unpublished. No system prompt: one response to each of 900 English questions, 30 questions per each of 30 traits. This describes default communication style and trait prominence, not objective answer quality or accuracy. No AA API, internal app-state access, automated collector, paid export or inference was used. Programmatically measured Words: Tokens that contain a letter or digit; each Chinese, Japanese or Korean character counts as one word (but all questions are in English). Source-native average over responses to the same900questions; heatmap colors use pool-relative percentiles, but the visible cell numbers here are native quantities, not those color percentiles. The Characters definition explicitly removes code and URLs; exact feature-specific handling follows the methodology. Average sentence length is words per sentence, not per response. Ends with a question is the share of responses (%); bold/italic count spans, not words or characters. Descriptive communication style, not a quality ranking. Original Flash and Flash1017 are distinct explicitly named checkpoints and retain separate IDs. Source label: 537.

1570 · raw 1570 Elo

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: Artificial Analysis published model evaluation. Native combined analytical-quality, presentation and rubric Elo; separate from old unversioned AA-Briefcase records. Manually checked rendered chart/summary, not an API import. Displayed rounding is preserved; no hidden precision, confidence interval, inference date or weight SHA is inferred. Source explicitly names1017 (Max), not the September model. AA advertises text/image input for its hosted evaluation; exact serving precision and visual pathway are unpublished.

47.5% · raw 47.5 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: Artificial Analysis published model evaluation. Task-level binary rubric pass rate; separate from analytical/presentation preference Elo. Manually checked rendered chart/summary, not an API import. Displayed rounding is preserved; no hidden precision, confidence interval, inference date or weight SHA is inferred. Source explicitly names1017 (Max), not the September model. AA advertises text/image input for its hosted evaluation; exact serving precision and visual pathway are unpublished.

1766 · raw 1766 Elo

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: Artificial Analysis published model evaluation. Analytical quality preference Elo, separate from presentation and combined Elo. Manually checked rendered chart/summary, not an API import. Displayed rounding is preserved; no hidden precision, confidence interval, inference date or weight SHA is inferred. Source explicitly names1017 (Max), not the September model. AA advertises text/image input for its hosted evaluation; exact serving precision and visual pathway are unpublished.

1602 · raw 1602 Elo

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: Artificial Analysis published model evaluation. Presentation preference Elo, separate from analytical quality and combined Elo. Manually checked rendered chart/summary, not an API import. Displayed rounding is preserved; no hidden precision, confidence interval, inference date or weight SHA is inferred. Source explicitly names1017 (Max), not the September model. AA advertises text/image input for its hosted evaluation; exact serving precision and visual pathway are unpublished.

5 · raw 5 index

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: Artificial Analysis published model evaluation. Signed knowledge reliability index, not percent accuracy; correct answers rewarded and hallucinations penalized. Rounded public chart observation. Manually checked rendered chart/summary, not an API import. Displayed rounding is preserved; no hidden precision, confidence interval, inference date or weight SHA is inferred. Source explicitly names1017 (Max), not the September model. AA advertises text/image input for its hosted evaluation; exact serving precision and visual pathway are unpublished.

43% · raw 43 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: Artificial Analysis published model evaluation. Share of all questions answered correctly in AA’s own knowledge evaluation. Rounded public chart label, not the vendor’s public-set test. Manually checked rendered chart/summary, not an API import. Displayed rounding is preserved; no hidden precision, confidence interval, inference date or weight SHA is inferred. Source explicitly names1017 (Max), not the September model. AA advertises text/image input for its hosted evaluation; exact serving precision and visual pathway are unpublished.

67% · raw 67 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: Artificial Analysis published model evaluation. Incorrect / (incorrect + partial + not attempted), as defined by the source. Lower is better; not overall error fraction. Manually checked rendered chart/summary, not an API import. Displayed rounding is preserved; no hidden precision, confidence interval, inference date or weight SHA is inferred. Source explicitly names1017 (Max), not the September model. AA advertises text/image input for its hosted evaluation; exact serving precision and visual pathway are unpublished.

67% · raw 67 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: Artificial Analysis published model evaluation. AA agentic SaaS workflow objective reward. Separate from a vendor’s strict all-assertion task pass rate. Manually checked rendered chart/summary, not an API import. Displayed rounding is preserved; no hidden precision, confidence interval, inference date or weight SHA is inferred. Source explicitly names1017 (Max), not the September model. AA advertises text/image input for its hosted evaluation; exact serving precision and visual pathway are unpublished.

43% · raw 43 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: Artificial Analysis published model evaluation. AA’s published Humanity’s Last Exam observation; exact tools/population details are not restated on the captured model chart. Manually checked rendered chart/summary, not an API import. Displayed rounding is preserved; no hidden precision, confidence interval, inference date or weight SHA is inferred. Source explicitly names1017 (Max), not the September model. AA advertises text/image input for its hosted evaluation; exact serving precision and visual pathway are unpublished.

17% · raw 17 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: Artificial Analysis published model evaluation. AA professional-document reasoning all-pass rate. Distinct from GDPval work-product Elo. Manually checked rendered chart/summary, not an API import. Displayed rounding is preserved; no hidden precision, confidence interval, inference date or weight SHA is inferred. Source explicitly names1017 (Max), not the September model. AA advertises text/image input for its hosted evaluation; exact serving precision and visual pathway are unpublished.

34% · raw 34 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: Artificial Analysis published model evaluation. AA’s Terminal-Bench4.0 model-plus-agent setup, separate from vendor harness observations. Manually checked rendered chart/summary, not an API import. Displayed rounding is preserved; no hidden precision, confidence interval, inference date or weight SHA is inferred. Source explicitly names1017 (Max), not the September model. AA advertises text/image input for its hosted evaluation; exact serving precision and visual pathway are unpublished.

83% · raw 83 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: Artificial Analysis published model evaluation. Version1.1 of AA long-context reasoning. Separate from old unversioned AA-LCR observations. Manually checked rendered chart/summary, not an API import. Displayed rounding is preserved; no hidden precision, confidence interval, inference date or weight SHA is inferred. Source explicitly names1017 (Max), not the September model. AA advertises text/image input for its hosted evaluation; exact serving precision and visual pathway are unpublished.

56% · raw 56 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: Artificial Analysis published model evaluation. AA scientist-authored coding evaluation. Source currently labels this benchmark under review; preserve that uncertainty. Manually checked rendered chart/summary, not an API import. Displayed rounding is preserved; no hidden precision, confidence interval, inference date or weight SHA is inferred. Source explicitly names1017 (Max), not the September model. AA advertises text/image input for its hosted evaluation; exact serving precision and visual pathway are unpublished.

24% · raw 24 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: Artificial Analysis published model evaluation. AA research-level physics evaluation. Source currently labels this benchmark under review; preserve that uncertainty. Manually checked rendered chart/summary, not an API import. Displayed rounding is preserved; no hidden precision, confidence interval, inference date or weight SHA is inferred. Source explicitly names1017 (Max), not the September model. AA advertises text/image input for its hosted evaluation; exact serving precision and visual pathway are unpublished.

81% · raw 81 %

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: Artificial Analysis published model evaluation. AA visual-reasoning observation for the exact named hosted model; serving precision, image pathway and execution date unpublished. Manually checked rendered chart/summary, not an API import. Displayed rounding is preserved; no hidden precision, confidence interval, inference date or weight SHA is inferred. Source explicitly names1017 (Max), not the September model. AA advertises text/image input for its hosted evaluation; exact serving precision and visual pathway are unpublished.

45 · raw 45 index

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: Artificial Analysis published model evaluation. AA Intelligence Index v4.3.2; public rounded summary. Exact measurement date and serving configuration remain unknown. Manually checked rendered chart/summary, not an API import. Displayed rounding is preserved; no hidden precision, confidence interval, inference date or weight SHA is inferred. Source explicitly names1017 (Max), not the September model. AA advertises text/image input for its hosted evaluation; exact serving precision and visual pathway are unpublished.

210.6 tok/s · raw 210.6 tok/s

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: Artificial Analysis published model evaluation. AA observed output speed, separate from OpenRouter endpoint measurements or vendor throughput. Manually checked rendered chart/summary, not an API import. Displayed rounding is preserved; no hidden precision, confidence interval, inference date or weight SHA is inferred. Source explicitly names1017 (Max), not the September model. AA advertises text/image input for its hosted evaluation; exact serving precision and visual pathway are unpublished.

$0.26 · raw 0.26 USD

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: Artificial Analysis published model evaluation. AA usage-priced task cost, not an API token tariff, invoice or subscription coefficient. Manually checked rendered chart/summary, not an API import. Displayed rounding is preserved; no hidden precision, confidence interval, inference date or weight SHA is inferred. Source explicitly names1017 (Max), not the September model. AA advertises text/image input for its hosted evaluation; exact serving precision and visual pathway are unpublished.

210 M tokens · raw 210 M tokens

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: Artificial Analysis published model evaluation. Source-rounded total output-token use for the AA Intelligence Index, in millions; not per-request output allowance. Manually checked rendered chart/summary, not an API import. Displayed rounding is preserved; no hidden precision, confidence interval, inference date or weight SHA is inferred. Source explicitly names1017 (Max), not the September model. AA advertises text/image input for its hosted evaluation; exact serving precision and visual pathway are unpublished. Summary uses a source-rounded millions-of-output-tokens unit.

48 · raw 48 index

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: Artificial Analysis published model evaluation. Manual native chart observation, current source industry aggregate; no computed average or hidden precision inferred. Components: 7 evaluations: AA-Omniscience, GDPval-AA v2.1, AA-Briefcase v1.1, Humanity's Last Exam, AutomationBench-AA, AA-LCR v1.1, GDP.pdf. Displayed index is not percent task success; current snapshot remains separate from older unversioned industry observations.

52 · raw 52 index

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: Artificial Analysis published model evaluation. Manual native chart observation, current source industry aggregate; no computed average or hidden precision inferred. Components: 6 evaluations: GDPval-AA v2.1, AA-Briefcase v1.1, AA-Omniscience, AutomationBench-AA, AA-LCR v1.1, GDP.pdf. Displayed index is not percent task success; current snapshot remains separate from older unversioned industry observations.

46 · raw 46 index

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: Artificial Analysis published model evaluation. Manual native chart observation, current source industry aggregate; no computed average or hidden precision inferred. Components: 7 evaluations: AA-Omniscience, GDPval-AA v2.1, AA-Briefcase v1.1, Humanity's Last Exam, AA-LCR v1.1, GDP.pdf, AutomationBench-AA. Displayed index is not percent task success; current snapshot remains separate from older unversioned industry observations.

45 · raw 45 index

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: Artificial Analysis published model evaluation. Manual native chart observation, current source industry aggregate; no computed average or hidden precision inferred. Components: 6 evaluations: AA-Omniscience, Humanity's Last Exam, CritPt, GDPval-AA v2.1, AA-Briefcase v1.1, Terminal-Bench 4.0. Displayed index is not percent task success; current snapshot remains separate from older unversioned industry observations.

51 · raw 51 index

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: Artificial Analysis published model evaluation. Manual native chart observation, current source industry aggregate; no computed average or hidden precision inferred. Components: 5 evaluations: AA-Omniscience, Humanity's Last Exam, GDPval-AA v2.1, AA-Briefcase v1.1, AA-LCR v1.1. Displayed index is not percent task success; current snapshot remains separate from older unversioned industry observations.

1746 · raw 1746 Elo

Headline · max effort · Independent evaluator

Source/record date: 2026-10-20

Manually observed public GDPval-AA v2.1 leaderboard table, full 298-configuration pool checked 20 October 2026. Exact source configuration: DeepSeek V4.1 Flash 1017 (Max); creator: DeepSeek; source rank 7; source release-month label Oct 2026 is model release metadata, not execution date. Published rounded native Elo 1746; literal displayed CI -21 / +21. Native Elo is not a percentage and is not the normalized (Elo−500)/2000 chart quantity. This versioned metric is separate from older generic GDPval-AA Elo and vendor GDPval/GDPval-AA results; no old evidence is overwritten. Execution date, inference precision, unprinted effort budgets and task costs are unpublished in this captured table. Elo is relative to the dated comparison pool; drift from an earlier observation does not prove a new evaluation run. Headline: highest published native Elo for this exact approved model; tied values prefer existing featured effort then source order.

42.7 fixes · raw 42.7 fixes

Headline · max effort · Independent evaluator

Source/record date: 2026-10-19

Harness: DeepSeek Harness / DeepSeek API. Mean of 7 runs (42, 41, 47, 47, 40, 43 and 39 of 105) in DeepSeek Harness, DeepSeek's own agent. The first Harness run read the answers from outside its folder and was voided. 5 runs ran in resumed segments after a billing cutoff; their time is active time only. Billed at DeepSeek's off-peak rate. Judged by Grok. DeepSeek-V4.1-Flash-1017 at effort max in DEEPSEEK HARNESS (dsh 0.2.1-rc.2, DeepSeek's own agent), on the same model id (deepseek-flash) and the same DeepSeek endpoint (Anthropic Messages, output_config.effort) as the Claude Code rows - the harness is the variable. MEAN OF 7 RUNS: 42, 41, 47, 47, 40, 43 and 39 of 105 (repo 1: 19, 18, 19, 20, 16, 20 and 21 of 45 - repo 2: 23, 23, 28, 27, 24, 23 and 18 of 60), mean 42.7. Genuine unplanted extras 44, 42, 40, 39, 46, 51 and 46; claimed-only fixes 1, 2, 0, 1, 3, 0 and 0. Active minutes 71.2, 60.8, 62.8, 69.0, 58.6, 51.1 and 61.0; cost $2.28, $1.67, $1.40, $1.74, $2.15, $1.98 and $1.68. *** AGAINST CLAUDE CODE *** the same model at max in Claude Code stands at 39.1 (its own row): +3.6 here. With runs this spread the gap is a lean, not an established difference. *** SETUP *** headless profile with a JSON event stream; web search and fetch switched off (no internet during a run, as for every harness here); subagents on as shipped; permission mode danger-full-access, the equivalent of every other harness's skip-permissions flag; a fresh dsh home per leg; the integrity warning appended to the task in every run (work only inside this folder; anything outside is cheating and every tool call is checked). *** THE FIRST HARNESS RUN WAS VOIDED *** at max it read the upstream project's git history on repo 1 and diffed it against its working copy; asked directly, it confirmed it. That run is excluded and was replaced. *** 5 OF 7 RUNS RAN IN SEGMENTS *** DeepSeek's balance-based concurrency cap (429) and then the prepaid balance running out (402) stopped those legs before they wrote a report; after a top-up each continued its own dsh session to the end (one leg also lost a response stream and continued again). Their time counts active work only, each segment from its first to its last successful model call; tokens and cost sum every segment. The resumed segments, and every later run, carried a retry policy for rate limits and dropped streams, which changes no model behaviour. *** EFFORT max *** read on the wire for every request: 7,108 of 7,123 requests carried max; the rest are dsh's own housekeeping calls (a no-effort side call per leg; one context-condensing call at high). *** COST IS A REAL BILL AT OFF-PEAK RATES *** DeepSeek's own per-call usage at its off-peak list price (a weekend); the account balance moved by the same amount within 1%. *** JUDGE *** Grok (grok-4.7). *** RECEIPTS *** 4 of the 14 legs shared the laptop with another arm and the rest had it to themselves, which the wall marker discloses per leg - scores, tokens and cost are untouched by that, only the clock. Native planted fixes out of 105, not percent; partial, claimed-only and unplanted extras are excluded. Aggregate dates do not establish every leg execution date. Same model with different coding agents: no checkpoint-only improvement or statistically established harness advantage is claimed. Overlapping sample-size means remain historical alternatives. Task bills and active-only timing are not token tariffs or calculator coefficients. Default: publisher-featured DeepSeek Harness max mean; effort preferences keep this same harness first.

39.2 fixes · raw 39.2 fixes

Alternative · high effort · Independent evaluator

Source/record date: 2026-10-19

Harness: DeepSeek Harness / DeepSeek API. Mean of 5 runs (45, 37, 41, 41 and 32 of 105) in DeepSeek Harness, DeepSeek's own agent. Billed at DeepSeek's off-peak rate. Judged by Grok. DeepSeek-V4.1-Flash-1017 at effort high in DEEPSEEK HARNESS (dsh 0.2.1-rc.2, DeepSeek's own agent), on the same model id (deepseek-flash) and the same DeepSeek endpoint (Anthropic Messages, output_config.effort) as the Claude Code rows - the harness is the variable. MEAN OF 5 RUNS: 45, 37, 41, 41 and 32 of 105 (repo 1: 19, 19, 15, 18 and 14 of 45 - repo 2: 26, 18, 26, 23 and 18 of 60), mean 39.2. Genuine unplanted extras 52, 58, 42, 36 and 31; claimed-only fixes 0, 2, 0, 0 and 0. Active minutes 49.3, 43.9, 69.7, 45.3 and 57.1; cost $1.41, $1.30, $1.09, $1.52 and $1.06. *** AGAINST CLAUDE CODE *** the same model at high in Claude Code stands at 36.4 (its own row): +2.8 here. With runs this spread the gap is a lean, not an established difference. *** SETUP *** headless profile with a JSON event stream; web search and fetch switched off (no internet during a run, as for every harness here); subagents on as shipped; permission mode danger-full-access, the equivalent of every other harness's skip-permissions flag; a fresh dsh home per leg; the integrity warning appended to the task in every run (work only inside this folder; anything outside is cheating and every tool call is checked). *** ONE SEGMENT EACH *** every leg ran straight through, with a retry policy for rate limits and dropped streams (changes no model behaviour). *** EFFORT high *** read on the wire for every request: 3,764 of 3,774 requests carried high; the rest are dsh's own housekeeping calls (a no-effort side call per leg). *** COST IS A REAL BILL AT OFF-PEAK RATES *** DeepSeek's own per-call usage at its off-peak list price (a weekend); the account balance moved by the same amount within 1%. *** JUDGE *** Grok (grok-4.7). *** RECEIPTS *** Every leg shared the laptop with another arm, which the wall marker discloses - scores, tokens and cost are untouched by that, only the clock. Native planted fixes out of 105, not percent; partial, claimed-only and unplanted extras are excluded. Aggregate dates do not establish every leg execution date. Same model with different coding agents: no checkpoint-only improvement or statistically established harness advantage is claimed. Overlapping sample-size means remain historical alternatives. Task bills and active-only timing are not token tariffs or calculator coefficients. Alternative coding-agent/effort observation; DeepSeek Harness max is the single default headline.

34.8 fixes · raw 34.8 fixes

Alternative · low effort · Independent evaluator

Source/record date: 2026-10-19

Harness: DeepSeek Harness / DeepSeek API. Mean of 5 runs (34, 36, 36, 34 and 34 of 105) in DeepSeek Harness, DeepSeek's own agent. Billed at DeepSeek's off-peak rate. Judged by Grok. DeepSeek-V4.1-Flash-1017 at effort low in DEEPSEEK HARNESS (dsh 0.2.1-rc.2, DeepSeek's own agent), on the same model id (deepseek-flash) and the same DeepSeek endpoint (Anthropic Messages, output_config.effort) as the Claude Code rows - the harness is the variable. MEAN OF 5 RUNS: 34, 36, 36, 34 and 34 of 105 (repo 1: 18, 17, 16, 16 and 15 of 45 - repo 2: 16, 19, 20, 18 and 19 of 60), mean 34.8. Genuine unplanted extras 38, 29, 40, 32 and 33; claimed-only fixes 0, 1, 0, 0 and 0. Active minutes 29.3, 31.5, 35.6, 38.7 and 33.9; cost $1.26, $0.93, $0.99, $1.03 and $1.10. *** AGAINST CLAUDE CODE *** the same model at low in Claude Code stands at 34.4 (its own row): +0.4 here. With runs this spread the gap is a lean, not an established difference. *** SETUP *** headless profile with a JSON event stream; web search and fetch switched off (no internet during a run, as for every harness here); subagents on as shipped; permission mode danger-full-access, the equivalent of every other harness's skip-permissions flag; a fresh dsh home per leg; the integrity warning appended to the task in every run (work only inside this folder; anything outside is cheating and every tool call is checked). *** ONE SEGMENT EACH *** every leg ran straight through, with a retry policy for rate limits and dropped streams (changes no model behaviour). *** EFFORT low *** read on the wire for every request: 3,719 of 3,729 requests carried low; the rest are dsh's own housekeeping calls (a no-effort side call per leg). *** COST IS A REAL BILL AT OFF-PEAK RATES *** DeepSeek's own per-call usage at its off-peak list price (a weekend); the account balance moved by the same amount within 1%. *** JUDGE *** Grok (grok-4.7). *** RECEIPTS *** Every leg shared the laptop with another arm, which the wall marker discloses - scores, tokens and cost are untouched by that, only the clock. Native planted fixes out of 105, not percent; partial, claimed-only and unplanted extras are excluded. Aggregate dates do not establish every leg execution date. Same model with different coding agents: no checkpoint-only improvement or statistically established harness advantage is claimed. Overlapping sample-size means remain historical alternatives. Task bills and active-only timing are not token tariffs or calculator coefficients. Alternative coding-agent/effort observation; DeepSeek Harness max is the single default headline.

42.2 fixes · raw 42.2 fixes

Alternative · max effort · Independent evaluator

Source/record date: 2026-10-19

Harness: Pi / DeepSeek API. Mean of 5 runs (41, 41, 43, 44 and 42 of 105) in Pi, the open-source minimal coding agent, through DeepSeek's OpenAI-compatible endpoint. Billed at DeepSeek's off-peak rate. Judged by Grok. DeepSeek-V4.1-Flash-1017 at effort max in PI (pi-coding-agent 1.3.0, Earendil's open-source minimal coding agent), on the same model id (deepseek-flash) and DeepSeek account as the Claude Code and DeepSeek Harness rows - the harness is the variable. Pi uses DeepSeek's OpenAI-compatible endpoint where the other two use the Anthropic-compatible one. MEAN OF 5 RUNS: 41, 41, 43, 44 and 42 of 105 (repo 1: 16, 15, 20, 17 and 21 of 45 - repo 2: 25, 26, 23, 27 and 21 of 60), mean 42.2. Genuine unplanted extras 34, 29, 48, 66 and 32; claimed-only fixes 0, 2, 1, 1 and 0. Minutes 37.7, 47.8, 66.3, 69.6 and 49.5; cost $0.54, $0.68, $1.03, $1.12 and $0.79. *** AGAINST THE OTHER HARNESSES *** the same model at max stands at 39.1 in Claude Code and 42.7 in DeepSeek Harness (means of 7, their own rows): +3.1 over Claude Code, at a lower cost per run than either. Pi runs no subagents, so whatever lifts it above Claude Code is not subagents. With runs this spread the gaps are a lean, not an established difference. *** SETUP *** print mode with a JSON event stream; built-in tools only (read, bash, edit, write) - Pi ships no subagents and no web tools; extensions, skills, MCP and prompt templates off; a fresh Pi config directory per leg; the run repo's own CLAUDE.md loads as Pi's context file; the integrity warning appended to the task in every run (work only inside this folder; anything outside is cheating and every tool call is checked). Every leg ran in one segment. *** EFFORT max *** read on the wire for every request: 1,902 of 1,902 carried reasoning_effort=max (Pi's catalog maps --thinking max to max for this model). *** COST IS A REAL BILL AT OFF-PEAK RATES *** DeepSeek's own per-call usage at its off-peak list price (a weekend); the account balance moved by the same amount. Planned as n=7, cut to n=5 for budget before the sixth run. *** JUDGE *** Grok (grok-4.7). *** RECEIPTS *** Every leg shared the laptop with another arm, which the wall marker discloses - scores, tokens and cost are untouched by that, only the clock. Native planted fixes out of 105, not percent; partial, claimed-only and unplanted extras are excluded. Aggregate dates do not establish every leg execution date. Same model with different coding agents: no checkpoint-only improvement or statistically established harness advantage is claimed. Overlapping sample-size means remain historical alternatives. Task bills and active-only timing are not token tariffs or calculator coefficients. Alternative coding-agent/effort observation; DeepSeek Harness max is the single default headline.

40.4 fixes · raw 40.4 fixes

Alternative · high effort · Independent evaluator

Source/record date: 2026-10-19

Harness: Pi / DeepSeek API. Mean of 5 runs (40, 39, 41, 38 and 44 of 105) in Pi, the open-source minimal coding agent, through DeepSeek's OpenAI-compatible endpoint. Billed at DeepSeek's off-peak rate. Judged by Grok. DeepSeek-V4.1-Flash-1017 at effort high in PI (pi-coding-agent 1.3.0, Earendil's open-source minimal coding agent), on the same model id (deepseek-flash) and DeepSeek account as the Claude Code and DeepSeek Harness rows - the harness is the variable. Pi uses DeepSeek's OpenAI-compatible endpoint where the other two use the Anthropic-compatible one. MEAN OF 5 RUNS: 40, 39, 41, 38 and 44 of 105 (repo 1: 18, 14, 15, 13 and 20 of 45 - repo 2: 22, 25, 26, 25 and 24 of 60), mean 40.4. Genuine unplanted extras 36, 34, 23, 17 and 37; claimed-only fixes 2, 0, 0, 0 and 2. Minutes 57.7, 51.8, 48.3, 40.8 and 43.1; cost $0.64, $0.83, $0.68, $0.44 and $0.62. *** AGAINST THE OTHER HARNESSES *** the same model at high stands at 36.4 in Claude Code and 39.2 in DeepSeek Harness (means of 5, their own rows); Pi at max: 42.2: +4.0 over Claude Code, at a lower cost per run than either. Pi runs no subagents, so whatever lifts it above Claude Code is not subagents. With runs this spread the gaps are a lean, not an established difference. *** SETUP *** print mode with a JSON event stream; built-in tools only (read, bash, edit, write) - Pi ships no subagents and no web tools; extensions, skills, MCP and prompt templates off; a fresh Pi config directory per leg; the run repo's own CLAUDE.md loads as Pi's context file; the integrity warning appended to the task in every run (work only inside this folder; anything outside is cheating and every tool call is checked). Every leg ran in one segment. *** EFFORT high *** read on the wire for every request: 1,637 of 1,637 carried reasoning_effort=high (Pi's catalog maps --thinking high to high for this model). *** COST IS A REAL BILL AT OFF-PEAK RATES *** DeepSeek's own per-call usage at its off-peak list price (a weekend); the account balance moved by the same amount. *** JUDGE *** Grok (grok-4.7). *** RECEIPTS *** Every leg shared the laptop with another arm, which the wall marker discloses - scores, tokens and cost are untouched by that, only the clock. Native planted fixes out of 105, not percent; partial, claimed-only and unplanted extras are excluded. Aggregate dates do not establish every leg execution date. Same model with different coding agents: no checkpoint-only improvement or statistically established harness advantage is claimed. Overlapping sample-size means remain historical alternatives. Task bills and active-only timing are not token tariffs or calculator coefficients. Alternative coding-agent/effort observation; DeepSeek Harness max is the single default headline.

33.8 fixes · raw 33.8 fixes

Alternative · low effort · Independent evaluator

Source/record date: 2026-10-19

Harness: Pi / DeepSeek API. Mean of 5 runs (37, 38, 32, 37 and 25 of 105) in Pi, the open-source minimal coding agent, through DeepSeek's OpenAI-compatible endpoint. Billed at DeepSeek's off-peak rate. Judged by Grok. DeepSeek-V4.1-Flash-1017 at effort low in PI (pi-coding-agent 1.3.0, Earendil's open-source minimal coding agent), on the same model id (deepseek-flash) and DeepSeek account as the Claude Code and DeepSeek Harness rows - the harness is the variable. Pi uses DeepSeek's OpenAI-compatible endpoint where the other two use the Anthropic-compatible one. MEAN OF 5 RUNS: 37, 38, 32, 37 and 25 of 105 (repo 1: 16, 15, 14, 14 and 11 of 45 - repo 2: 21, 23, 18, 23 and 14 of 60), mean 33.8. Genuine unplanted extras 21, 16, 16, 26 and 19; claimed-only fixes 3, 0, 2, 0 and 0. Minutes 39.4, 31.2, 28.8, 34.6 and 32.1; cost $0.54, $0.41, $0.35, $0.53 and $0.46. *** AGAINST THE OTHER HARNESSES *** the same model at low stands at 34.4 in Claude Code and 34.8 in DeepSeek Harness (means of 5, their own rows); Pi at high: 40.4, at max: 42.2: -0.6 over Claude Code, at a lower cost per run than either. Pi runs no subagents, so whatever lifts it above Claude Code is not subagents. With runs this spread the gaps are a lean, not an established difference. *** SETUP *** print mode with a JSON event stream; built-in tools only (read, bash, edit, write) - Pi ships no subagents and no web tools; extensions, skills, MCP and prompt templates off; a fresh Pi config directory per leg; the run repo's own CLAUDE.md loads as Pi's context file; the integrity warning appended to the task in every run (work only inside this folder; anything outside is cheating and every tool call is checked). Every leg ran in one segment. *** EFFORT low *** read on the wire for every request: 1,204 of 1,204 carried reasoning_effort=low (Pi's catalog maps --thinking low to low for this model). Run 5 scored 25 in a normal run (about 16 minutes per repo, 125 and 180 tool calls) - a weak run, not a failure; the median of the five is 37. *** COST IS A REAL BILL AT OFF-PEAK RATES *** DeepSeek's own per-call usage at its off-peak list price (a weekend); the account balance moved by the same amount. *** JUDGE *** Grok (grok-4.7). *** RECEIPTS *** Every leg shared the laptop with another arm, which the wall marker discloses - scores, tokens and cost are untouched by that, only the clock. Native planted fixes out of 105, not percent; partial, claimed-only and unplanted extras are excluded. Aggregate dates do not establish every leg execution date. Same model with different coding agents: no checkpoint-only improvement or statistically established harness advantage is claimed. Overlapping sample-size means remain historical alternatives. Task bills and active-only timing are not token tariffs or calculator coefficients. Alternative coding-agent/effort observation; DeepSeek Harness max is the single default headline.

39.1 fixes · raw 39.1 fixes

Alternative · max effort · Independent evaluator

Source/record date: 2026-10-19

Harness: Claude Code / DeepSeek API. Mean of 7 runs (34, 37, 45, 46, 35, 39 and 38 of 105); runs that read the answers from outside their folder were voided; 4 runs carried an integrity warning. The same model in DeepSeek Harness is its own row. Billed at DeepSeek's off-peak rate. Judged by Grok. DeepSeek-V4.1-Flash-1017 at effort max on DeepSeek's own Anthropic-compatible API in Claude Code. MEAN OF 7 RUNS: 34, 37, 45, 46, 35, 39 and 38 of 105 (repo 1: 17, 19, 17, 19, 19, 19 and 17 of 45 - repo 2: 17, 18, 28, 27, 16, 20 and 21 of 60), mean 39.1. Genuine unplanted extras 19, 16, 32, 37, 19, 22 and 15; claimed-only fixes 2, 5, 0, 1, 0, 1 and 1. Minutes 50.6, 54.1, 57.5, 90.2, 56.4, 45.0 and 50.0; cost $0.90, $0.98, $1.13, $1.43, $0.83, $0.82 and $0.99. *** THE SAME MODEL IN DEEPSEEK HARNESS *** (DeepSeek's own agent, same model id and endpoint) is its own row: mean of 7 at max, 42.7. *** RUNS THAT READ THE ANSWERS WERE VOIDED *** Across this model's runs, three legs found material outside their own folder that reveals the planted bugs - the upstream project's git history on this machine, or installed and downloaded release builds of the same extension - and diffed it against their working copy. Every leg's tool calls were checked; those runs are excluded, and each one, asked directly, confirmed it. *** 4 OF THESE RUNS CARRIED AN INTEGRITY WARNING *** appended to the untouched task after that was found: work only inside this folder; anything outside is cheating and every tool call is checked. The other 3 ran the standard task. *** EFFORT max *** the top tier DeepSeek lists for the model; the endpoint accepted it and rejected an invented tier before the runs. *** COST IS A REAL BILL AT OFF-PEAK RATES *** a weekend, DeepSeek's off-peak rate all day; runs that shared the key at the same time split the balance drop by each leg's token cost at DeepSeek's list price. *** JUDGE *** Grok (grok-4.7). *** RECEIPTS *** 12 of the 14 legs shared the laptop with another arm and the rest had it to themselves, which the wall marker discloses per leg - scores, tokens and cost are untouched by that, only the clock. Native planted fixes out of 105, not percent; partial, claimed-only and unplanted extras are excluded. Aggregate dates do not establish every leg execution date. Same model with different coding agents: no checkpoint-only improvement or statistically established harness advantage is claimed. Overlapping sample-size means remain historical alternatives. Task bills and active-only timing are not token tariffs or calculator coefficients. Alternative coding-agent/effort observation; DeepSeek Harness max is the single default headline.

39.4 fixes · raw 39.4 fixes

Alternative · max effort · Independent evaluator

Source/record date: 2026-10-18

Harness: Claude Code / DeepSeek API. Mean of 5 runs (34, 37, 45, 46 and 35 of 105); runs that read the answers from outside their folder were voided; 2 runs carried an integrity warning. Billed at DeepSeek's off-peak rate (a Saturday). Judged by Grok. DeepSeek-V4.1-Flash-1017 at effort max on DeepSeek's own Anthropic-compatible API in Claude Code. MEAN OF 5 RUNS: 34, 37, 45, 46 and 35 of 105 (repo 1: 17, 19, 17, 19 and 19 of 45 - repo 2: 17, 18, 28, 27 and 16 of 60), mean 39.4. Genuine unplanted extras 19, 16, 32, 37 and 19; claimed-only fixes 2, 5, 0, 1 and 0. Minutes 50.6, 54.1, 57.5, 90.2 and 56.4; cost $0.90, $0.98, $1.13, $1.43 and $0.83. *** RUNS THAT READ THE ANSWERS WERE VOIDED *** Across this model's runs, three legs found material outside their own folder that reveals the planted bugs - the upstream project's git history on this machine, or installed and downloaded release builds of the same extension - and diffed it against their working copy. Every leg's tool calls were checked; those runs are excluded, and each one, asked directly, confirmed it. *** 2 OF THESE RUNS CARRIED AN INTEGRITY WARNING *** appended to the untouched task after that was found: work only inside this folder; anything outside is cheating and every tool call is checked. The other 3 ran the standard task. *** EFFORT max *** a tier DeepSeek lists for the model; the endpoint accepted it and rejected an invented tier before the runs. *** COST IS A REAL BILL AT OFF-PEAK RATES *** a Saturday, half DeepSeek's peak rate all day; runs that shared the key at the same time split the balance drop by each leg's token cost at DeepSeek's list price. *** JUDGE *** Grok (grok-4.7). *** RECEIPTS *** 8 of the 10 legs shared the laptop with another arm and the rest had it to themselves, which the wall marker discloses per leg - scores, tokens and cost are untouched by that, only the clock. Published fixes are native counts, not percent. Individual leg execution dates are not established by the aggregate row date. These sample-size observations overlap earlier receipts; they are not additional independent benchmarks. The judge differs from September’s Codex GPT-5.5, so no controlled checkpoint-only gain is claimed. Bills use Saturday off-peak rates and shared-key token-share allocation; task bills are not token tariffs. Current publisher-featured five-run max headline. Demoted 2026-10-19: retained Claude Code five-run receipt. The current same-agent seven-run mean is 39.1; the publisher-featured default 42.7 uses DeepSeek Harness, a different coding agent. No model-only gain is implied.

36.4 fixes · raw 36.4 fixes

Alternative · high effort · Independent evaluator

Source/record date: 2026-10-18

Harness: Claude Code / DeepSeek API. Mean of 5 runs (34, 30, 42, 38 and 38 of 105); runs that read the answers from outside their folder were voided; 3 runs carried an integrity warning. Billed at DeepSeek's off-peak rate (a Saturday). Judged by Grok. DeepSeek-V4.1-Flash-1017 at effort high on DeepSeek's own Anthropic-compatible API in Claude Code. MEAN OF 5 RUNS: 34, 30, 42, 38 and 38 of 105 (repo 1: 15, 13, 16, 13 and 19 of 45 - repo 2: 19, 17, 26, 25 and 19 of 60), mean 36.4. Genuine unplanted extras 18, 18, 29, 37 and 20; claimed-only fixes 1, 3, 1, 0 and 3. Minutes 47.2, 41.7, 56.4, 70.6 and 56.6; cost $0.87, $0.71, $0.86, $1.02 and $0.64. *** RUNS THAT READ THE ANSWERS WERE VOIDED *** Across this model's runs, three legs found material outside their own folder that reveals the planted bugs - the upstream project's git history on this machine, or installed and downloaded release builds of the same extension - and diffed it against their working copy. Every leg's tool calls were checked; those runs are excluded, and each one, asked directly, confirmed it. *** 3 OF THESE RUNS CARRIED AN INTEGRITY WARNING *** appended to the untouched task after that was found: work only inside this folder; anything outside is cheating and every tool call is checked. The other 2 ran the standard task. *** EFFORT high *** a tier DeepSeek lists for the model; the endpoint accepted it and rejected an invented tier before the runs. *** COST IS A REAL BILL AT OFF-PEAK RATES *** a Saturday, half DeepSeek's peak rate all day; runs that shared the key at the same time split the balance drop by each leg's token cost at DeepSeek's list price. *** JUDGE *** Grok (grok-4.7). *** RECEIPTS *** Every leg shared the laptop with another arm, which the wall marker discloses - scores, tokens and cost are untouched by that, only the clock. Published fixes are native counts, not percent. Individual leg execution dates are not established by the aggregate row date. These sample-size observations overlap earlier receipts; they are not additional independent benchmarks. The judge differs from September’s Codex GPT-5.5, so no controlled checkpoint-only gain is claimed. Bills use Saturday off-peak rates and shared-key token-share allocation; task bills are not token tariffs. Retained effort-specific five-run mean; max is the one default headline.

34.4 fixes · raw 34.4 fixes

Alternative · low effort · Independent evaluator

Source/record date: 2026-10-18

Harness: Claude Code / DeepSeek API. Mean of 5 runs (40, 34, 32, 33 and 33 of 105); runs that read the answers from outside their folder were voided; 3 runs carried an integrity warning. Billed at DeepSeek's off-peak rate (a Saturday). Judged by Grok. DeepSeek-V4.1-Flash-1017 at effort low on DeepSeek's own Anthropic-compatible API in Claude Code. MEAN OF 5 RUNS: 40, 34, 32, 33 and 33 of 105 (repo 1: 18, 15, 16, 16 and 17 of 45 - repo 2: 22, 19, 16, 17 and 16 of 60), mean 34.4. Genuine unplanted extras 27, 10, 16, 21 and 10; claimed-only fixes 1, 0, 1, 1 and 3. Minutes 49.6, 45.4, 38.5, 55.4 and 33.3; cost $0.80, $0.71, $0.67, $0.83 and $0.54. *** RUNS THAT READ THE ANSWERS WERE VOIDED *** Across this model's runs, three legs found material outside their own folder that reveals the planted bugs - the upstream project's git history on this machine, or installed and downloaded release builds of the same extension - and diffed it against their working copy. Every leg's tool calls were checked; those runs are excluded, and each one, asked directly, confirmed it. *** 3 OF THESE RUNS CARRIED AN INTEGRITY WARNING *** appended to the untouched task after that was found: work only inside this folder; anything outside is cheating and every tool call is checked. The other 2 ran the standard task. *** EFFORT low *** a tier DeepSeek lists for the model; the endpoint accepted it and rejected an invented tier before the runs. *** COST IS A REAL BILL AT OFF-PEAK RATES *** a Saturday, half DeepSeek's peak rate all day; runs that shared the key at the same time split the balance drop by each leg's token cost at DeepSeek's list price. *** JUDGE *** Grok (grok-4.7). *** RECEIPTS *** Every leg shared the laptop with another arm, which the wall marker discloses - scores, tokens and cost are untouched by that, only the clock. Published fixes are native counts, not percent. Individual leg execution dates are not established by the aggregate row date. These sample-size observations overlap earlier receipts; they are not additional independent benchmarks. The judge differs from September’s Codex GPT-5.5, so no controlled checkpoint-only gain is claimed. Bills use Saturday off-peak rates and shared-key token-share allocation; task bills are not token tariffs. Retained effort-specific five-run mean; max is the one default headline.

32 fixes · raw 32 fixes

Alternative · high effort · Independent evaluator

Source/record date: 2026-10-18

Harness: Claude Code / DeepSeek API. Mean of two runs (34 and 30 of 105); a third run that read the upstream project's git history was voided. Billed at DeepSeek's off-peak rate (a Saturday). Judged by Grok. DeepSeek-V4.1-Flash-1017 at effort high - the tier below max on DeepSeek's list - on DeepSeek's own Anthropic-compatible API in Claude Code, set up exactly as the max row except the effort string. MEAN OF TWO RUNS: 34 and 30 of 105 (repo 1: 15 and 13 of 45 - repo 2: 19 and 17 of 60), mean 32.0. Genuine unplanted extras 18 and 18; claimed-only fixes 1 and 3. Minutes 47.2 and 41.7; cost $0.87 and $0.71. Max stands at 38.7 (mean of three). *** A THIRD RUN WAS VOIDED *** It scored 45 of 45 on repo 1, and its transcript shows why: it listed the machine's other repositories, found the real upstream project the benchmark was built from, extracted the matching release from its git history and compared it with its run folder - which reveals every planted bug. That is not a measurement, so the whole run is excluded. The runs in this row listed the same folders but read nothing outside their own. *** EFFORT high *** a tier DeepSeek lists for the model; the endpoint accepted it and rejected an invented tier before every run. *** COST IS A REAL BILL AT OFF-PEAK RATES *** a Saturday, half DeepSeek's peak rate all day. The high runs ran at the same time on one key, so their combined bill was split between them by each leg's token cost at DeepSeek's list price. *** JUDGE *** Grok (grok-4.7). *** RECEIPTS *** Every leg shared the laptop with another arm, which the wall marker discloses - scores, tokens and cost are untouched by that, only the clock. Published fixes are native counts, not percent. Individual leg execution dates are not established by the aggregate row date. These sample-size observations overlap earlier receipts; they are not additional independent benchmarks. The judge differs from September’s Codex GPT-5.5, so no controlled checkpoint-only gain is claimed. Bills use Saturday off-peak rates and shared-key token-share allocation; task bills are not token tariffs. Historical two-run high observation; superseded by the five-run high mean.

38.7 fixes · raw 38.7 fixes

Alternative · max effort · Independent evaluator

Source/record date: 2026-10-17

Harness: Claude Code / DeepSeek API. DeepSeek-V4.1-Flash-1017 at max effort on DeepSeek's own Anthropic-compatible API in Claude Code, set up exactly as the earlier V4.1 Flash rows. MEAN OF 3 RUNS: 34, 37 and 45 of 105 (repo 1: 17, 19 and 17 of 45 - repo 2: 17, 18 and 28 of 60), mean 38.7. Genuine unplanted extras 19, 16 and 32; claimed-only fixes 2, 5 and 0. Minutes 50.6, 54.1 and 57.5; cost $0.90, $0.98 and $1.13. The previous V4.1 Flash checkpoint at the same rung stands at 21.7 (mean of three). *** SAME MODEL ID, NEW CHECKPOINT *** DeepSeek serves 1017 under the id it used before, deepseek-flash; its pricing page maps that id to DeepSeek-V4.1-Flash-1017 and routes the older Flash ids to it. *** EFFORT max *** the top tier DeepSeek lists for the model; the endpoint accepted max and rejected an invented tier with its accept-list before every run. *** COST IS A REAL BILL AT OFF-PEAK RATES *** all runs fell on a Saturday, which DeepSeek prices at its off-peak rate all day - half its peak rate. Run 1 is the prepaid balance drop around each leg. Runs 2 and 3 ran at the same time on one key, so their combined bill ($2.11) is the balance drop across both, split between them in proportion to each leg's token cost at DeepSeek's list price. The earlier V4.1 Flash rows were billed on a weekday. Run 1's two leg readings ($0.90) were taken before DeepSeek's balance finished settling: the balance around the whole of run 1 fell $1.00. The runner now waits for the balance to settle. *** A DIFFERENT JUDGE *** judged by Grok (grok-4.7), not Codex gpt-5.5 like the earlier DeepSeek rows: Codex has refused gpt-5.5 for the account since Oct 14. *** RECEIPTS *** 4 of the 6 legs shared the laptop with another arm and the rest had it to themselves, which the wall marker discloses per leg - scores, tokens and cost are untouched by that, only the clock. Mean of 3 runs (34, 37 and 45 of 105). Billed at DeepSeek's off-peak rate (a Saturday), about half the peak price; runs 2 and 3 shared one key and their bill is split by token share. Judged by Grok, not Codex like the earlier DeepSeek rows. Only planted bug fixes count; partial, claimed-only and genuine unplanted extras are excluded. Published scores use native fixes, not percentages; aggregate means retain the source precision. Judge changed from Codex gpt-5.5 on the September checkpoint to Grok grok-4.7 here; the score difference is not a controlled checkpoint-only gain. The source row date is retained without inventing individual leg execution dates. Headline: publisher-featured current three-run mean (34, 37, 45), ahead of the retained two-run mean and first run in source order. Superseded 2026-10-18: publisher now features the five-run max mean 39.4/105. This three-run receipt remains unchanged historical evidence; later accepted runs used a mix of standard and added integrity-warning prompts.

35.5 fixes · raw 35.5 fixes

Alternative · max effort · Independent evaluator

Source/record date: 2026-10-17

Harness: Claude Code / DeepSeek API. DeepSeek-V4.1-Flash-1017 at max effort on DeepSeek's own Anthropic-compatible API in Claude Code, set up exactly as the earlier V4.1 Flash rows. MEAN OF 2 RUNS: 34 and 37 of 105 (repo 1: 17 and 19 of 45 - repo 2: 17 and 18 of 60), mean 35.5. Genuine unplanted extras 19 and 16; claimed-only fixes 2 and 5. Minutes 50.6 and 54.1; cost $0.90 and $0.98. The previous V4.1 Flash checkpoint at the same rung stands at 21.7 (mean of three). *** SAME MODEL ID, NEW CHECKPOINT *** DeepSeek serves 1017 under the id it used before, deepseek-flash; its pricing page maps that id to DeepSeek-V4.1-Flash-1017 and routes the older Flash ids to it. *** EFFORT max *** the top tier DeepSeek lists for the model; the endpoint accepted max and rejected an invented tier with its accept-list before every run. *** COST IS A REAL BILL AT OFF-PEAK RATES *** all runs fell on a Saturday, which DeepSeek prices at its off-peak rate all day - half its peak rate. Run 1 is the prepaid balance drop around each leg. Runs 2 and 3 ran at the same time on one key, so their combined bill ($2.11) is the balance drop across both, split between them in proportion to each leg's token cost at DeepSeek's list price. The earlier V4.1 Flash rows were billed on a weekday. Run 1's two leg readings ($0.90) were taken before DeepSeek's balance finished settling: the balance around the whole of run 1 fell $1.00. The runner now waits for the balance to settle. *** A DIFFERENT JUDGE *** judged by Grok (grok-4.7), not Codex gpt-5.5 like the earlier DeepSeek rows: Codex has refused gpt-5.5 for the account since Oct 14. *** RECEIPTS *** 2 of the 4 legs shared the laptop with another arm and the rest had it to themselves, which the wall marker discloses per leg - scores, tokens and cost are untouched by that, only the clock. Mean of 2 runs (34 and 37 of 105). Billed at DeepSeek's off-peak rate (a Saturday), about half the peak price; runs 2 and 3 shared one key and their bill is split by token share. Judged by Grok, not Codex like the earlier DeepSeek rows. Only planted bug fixes count; partial, claimed-only and genuine unplanted extras are excluded. Published scores use native fixes, not percentages; aggregate means retain the source precision. Judge changed from Codex gpt-5.5 on the September checkpoint to Grok grok-4.7 here; the score difference is not a controlled checkpoint-only gain. The source row date is retained without inventing individual leg execution dates. Superseded 2026-10-17 by the mean of 3 runs (38.7 of 105). Nothing here is retracted - it is the receipt for what the board showed before the next run landed. Retained as a historical sample-size alternative; the three-run mean is the current headline.

34 fixes · raw 34 fixes

Alternative · max effort · Independent evaluator

Source/record date: 2026-10-17

Harness: Claude Code / DeepSeek API. DeepSeek-V4.1-Flash-1017, released 2026-10-17, at max effort on DeepSeek's own Anthropic-compatible API in Claude Code, set up exactly as the earlier V4.1 Flash rows. ONE RUN: 34 of 105 - 17 of 45 on repo 1 and 17 of 60 on repo 2 - with 2 partial fix(es), 2 claimed-only and 19 genuine unplanted extras; 50.6 minutes, $0.90. The previous V4.1 Flash checkpoint at the same rung stands at 21.7 (mean of three, $0.78). Two more runs are under way. *** SAME MODEL ID, NEW CHECKPOINT *** DeepSeek serves 1017 under the id it used before, deepseek-flash; its pricing page maps that id to DeepSeek-V4.1-Flash-1017 and says the older Flash ids are retired and routed to it. *** EFFORT max *** the top tier DeepSeek lists for the model (low, high, max); the endpoint accepted max and rejected an invented tier with an error naming its accept-list, checked before the legs ran. *** COST IS A REAL BILL AT OFF-PEAK RATES *** the prepaid balance drop around each leg. The run fell on a Saturday, which DeepSeek prices at its off-peak rate all day - half its peak rate - so this bill is about half what the same tokens cost at peak. The earlier V4.1 Flash rows were billed on a weekday. *** A DIFFERENT JUDGE *** judged by Grok (grok-4.7), not Codex gpt-5.5 like the earlier DeepSeek rows: Codex has refused gpt-5.5 for the account since Oct 14. Grok is this board's second calibrated judge and is not a sibling of DeepSeek, but scores from two judges are not perfectly interchangeable. *** RECEIPTS *** The machine was otherwise idle for every leg, so the wall figures compare directly against other uncontended rows. One run; two more are under way. Billed at DeepSeek's off-peak rate (a Saturday), about half the peak price. Judged by Grok, not Codex like the earlier DeepSeek rows. Only planted bug fixes count; partial, claimed-only and genuine unplanted extras are excluded. Published scores use native fixes, not percentages; aggregate means retain the source precision. Judge changed from Codex gpt-5.5 on the September checkpoint to Grok grok-4.7 here; the score difference is not a controlled checkpoint-only gain. The source row date is retained without inventing individual leg execution dates. Superseded 2026-10-17 by the mean of 2 runs (35.5 of 105). Nothing here is retracted - it is the receipt for what the board showed before the next run landed. Retained as a historical sample-size alternative; the three-run mean is the current headline.

37.6 fixes · raw 37.6 fixes

Alternative · max effort · Independent evaluator

Source/record date: 2026-10-20

Harness: OpenCode 1.19.3 / DeepSeek API. DeepSeek-V4.1-Flash-1017 at effort max in OPENCODE (opencode-ai 1.19.3, the open-source coding agent), on the same model id (deepseek-flash) and DeepSeek account as the Claude Code, DeepSeek Harness and Pi rows - the harness is the variable. OpenCode uses DeepSeek's OpenAI-compatible endpoint, like Pi. MEAN OF 5 RUNS: 36, 40, 35, 38 and 39 of 105 (repo 1: 13, 17, 18, 15 and 16 of 45 - repo 2: 23, 23, 17, 23 and 23 of 60), mean 37.6. Genuine unplanted extras 33, 37, 43, 36 and 32; claimed-only fixes 1, 0, 0, 0 and 0. Minutes 58.8, 41.9, 73.1, 49.8 and 55.1; cost $1.01, $0.75, $1.23, $0.76 and $1.04. *** AGAINST THE OTHER HARNESSES *** the same model at max stands at 39.1 in Claude Code, 42.7 in DeepSeek Harness and 42.2 in Pi (means of 7, 7 and 5, their own rows). OpenCode's best run (40) sits below every Pi run (41-44) and at the bottom of DeepSeek Harness's range (39-47), so at max it trails both; against Claude Code (34-46) the ranges overlap and the gap is a lean. *** SETUP *** `opencode run` with a JSON event stream; tools as shipped, including its task tool for subagents (it started 1 subagent across the ten legs, 154 of the 2,382 requests), except web fetch and web search, which are off; a fresh OpenCode config and data home per leg; its Claude-Code compatibility layer off and the run repo's own CLAUDE.md loaded explicitly as an instruction file; the integrity warning appended to the task in every run (work only inside this folder; anything outside is cheating and every tool call is checked). Every leg ran in one segment. OpenCode's stream reports no context accounting, but the tap's per-request prompt sizes show the main loop's context only ever grew (peak 803K tokens of the 1M window), so no leg compacted. Generated test-coverage reports were left out of the judged diff (generated output, never a fix). A first attempt at max was discarded unjudged - a launch fault made OpenCode work outside the run folders - and all ten legs were rerun with the fault fixed; these are the rerun's numbers. *** EFFORT max *** read on the wire: 2,372 of 2,382 requests carried reasoning_effort=max; the other 10 were OpenCode's own short session-title calls, which it sends at low whatever the variant. *** COST IS A REAL BILL AT OFF-PEAK RATES *** DeepSeek's own per-call usage at its off-peak list price (the runs fell in its off-peak window: a Sunday); the account balance moved by the same amount. *** JUDGE *** Grok (grok-4.7). *** RECEIPTS *** Every leg shared the laptop with another arm, which the wall marker discloses - scores, tokens and cost are untouched by that, only the clock. Native planted fixes out of105, not percent. Five-run aggregate is an alternative coding-agent setup, not a checkpoint improvement. All original harness headlines and effort preference order are preserved. Source publication date does not establish every leg execution date. Off-peak task bills and shared-machine wall time are not token tariffs or isolated speed measurements.

36.6 fixes · raw 36.6 fixes

Alternative · high effort · Independent evaluator

Source/record date: 2026-10-20

Harness: OpenCode 1.19.3 / DeepSeek API. DeepSeek-V4.1-Flash-1017 at effort high in OPENCODE (opencode-ai 1.19.3, the open-source coding agent), on the same model id (deepseek-flash) and DeepSeek account as the Claude Code, DeepSeek Harness and Pi rows - the harness is the variable. OpenCode uses DeepSeek's OpenAI-compatible endpoint, like Pi. MEAN OF 5 RUNS: 35, 37, 43, 28 and 40 of 105 (repo 1: 14, 15, 18, 14 and 16 of 45 - repo 2: 21, 22, 25, 14 and 24 of 60), mean 36.6. Genuine unplanted extras 11, 45, 24, 19 and 29; claimed-only fixes 0, 1, 1, 1 and 2. Minutes 28.7, 63.7, 53.6, 25.5 and 36.7; cost $0.43, $1.12, $1.17, $0.53 and $0.69. *** AGAINST THE OTHER HARNESSES *** the same model at high stands at 36.4 in Claude Code, 39.2 in DeepSeek Harness and 40.4 in Pi (means of 5, their own rows). OpenCode is level with Claude Code and below the other two, but its runs (28-43) overlap all three ranges (Claude Code 30-42, DeepSeek Harness 32-45, Pi 38-44), so at high the gaps are a lean, not an established difference. *** SETUP *** `opencode run` with a JSON event stream; tools as shipped, including its task tool for subagents (it started 8 subagents across the ten legs, 726 of the 2,241 requests), except web fetch and web search, which are off; a fresh OpenCode config and data home per leg; its Claude-Code compatibility layer off and the run repo's own CLAUDE.md loaded explicitly as an instruction file; the integrity warning appended to the task in every run (work only inside this folder; anything outside is cheating and every tool call is checked). Every leg ran in one segment. OpenCode's stream reports no context accounting, but the tap's per-request prompt sizes show the main loop's context only ever grew (peak 669K tokens of the 1M window), so no leg compacted. Generated test-coverage reports were left out of the judged diff (generated output, never a fix). *** EFFORT high *** read on the wire: 2,231 of 2,241 requests carried reasoning_effort=high; the other 10 were OpenCode's own short session-title calls, which it sends at low whatever the variant. *** COST IS A REAL BILL AT OFF-PEAK RATES *** DeepSeek's own per-call usage at its off-peak list price (the runs fell in its off-peak window: late Sunday UTC); the account balance moved by the same amount. *** JUDGE *** Grok (grok-4.7). *** RECEIPTS *** Every leg shared the laptop with another arm, which the wall marker discloses - scores, tokens and cost are untouched by that, only the clock. Native planted fixes out of105, not percent. Five-run aggregate is an alternative coding-agent setup, not a checkpoint improvement. All original harness headlines and effort preference order are preserved. Source publication date does not establish every leg execution date. Off-peak task bills and shared-machine wall time are not token tariffs or isolated speed measurements.

32.8 fixes · raw 32.8 fixes

Alternative · low effort · Independent evaluator

Source/record date: 2026-10-20

Harness: OpenCode 1.19.3 / DeepSeek API. DeepSeek-V4.1-Flash-1017 at effort low in OPENCODE (opencode-ai 1.19.3, the open-source coding agent), on the same model id (deepseek-flash) and DeepSeek account as the Claude Code, DeepSeek Harness and Pi rows - the harness is the variable. OpenCode uses DeepSeek's OpenAI-compatible endpoint, like Pi. MEAN OF 5 RUNS: 36, 32, 27, 31 and 38 of 105 (repo 1: 22, 12, 12, 13 and 20 of 45 - repo 2: 14, 20, 15, 18 and 18 of 60), mean 32.8. Genuine unplanted extras 25, 12, 9, 9 and 33; claimed-only fixes 1, 1, 1, 1 and 4. Minutes 49.0, 31.9, 26.2, 31.3 and 67.6; cost $0.90, $0.38, $0.41, $0.53 and $1.10. *** AGAINST THE OTHER HARNESSES *** the same model at low stands at 34.4 in Claude Code, 34.8 in DeepSeek Harness and 33.8 in Pi (means of 5, their own rows). OpenCode is the lowest of the four, but its runs (27-38) overlap all three ranges (Claude Code 32-40, DeepSeek Harness 34-36, Pi 25-38), so at low the gaps are a lean, not an established difference. *** SETUP *** `opencode run` with a JSON event stream; tools as shipped, including its task tool for subagents (it started 13 subagents across the ten legs, 1,030 of the 2,465 requests), except web fetch and web search, which are off; a fresh OpenCode config and data home per leg; its Claude-Code compatibility layer off and the run repo's own CLAUDE.md loaded explicitly as an instruction file; the integrity warning appended to the task in every run (work only inside this folder; anything outside is cheating and every tool call is checked). Every leg ran in one segment. OpenCode's stream reports no context accounting, but the tap's per-request prompt sizes show the main loop's context only ever grew (peak 515K tokens of the 1M window), so no leg compacted. Generated test-coverage reports were left out of the judged diff (generated output, never a fix). One leg (run 2, repo 1) was flagged by the automated integrity scan for a path string and cleared on review: the string came from the repo's own test fixtures, which the agent was parsing inside its folder. *** EFFORT low *** read on the wire: 2,465 of 2,465 requests carried reasoning_effort=low (the session-title calls included). *** COST IS A REAL BILL AT OFF-PEAK RATES *** DeepSeek's own per-call usage at its off-peak list price (the runs fell in its off-peak window: late Sunday into Monday 00:23 UTC, before the 01:00 peak); the account balance moved by the same amount. *** JUDGE *** Grok (grok-4.7). *** RECEIPTS *** Every leg shared the laptop with another arm, which the wall marker discloses - scores, tokens and cost are untouched by that, only the clock. Native planted fixes out of105, not percent. Five-run aggregate is an alternative coding-agent setup, not a checkpoint improvement. All original harness headlines and effort preference order are preserved. Source publication date does not establish every leg execution date. Off-peak task bills and shared-machine wall time are not token tariffs or isolated speed measurements.