AutomationBench v1.0.6 Results Across LLMs

54.8% · raw 54.8 %

Headline · unknown effort · Third-party

Source/record date: 2026-09-10

VentureBeat quoting DeepSeek

53.1% · raw 53.1 %

Headline · unknown effort · Own vendor

Source/record date: 2026-09-21

52.5% · raw 52.5 %

Headline · unknown effort · Third-party

Source/record date: 2026-09-30

Via ThreatFrontier transcription of Ant launch table; original pixels not yet re-read. Effort undisclosed.

52.3% · raw 52.3 %

Headline · unknown effort · Own vendor

Source/record date: 2026-09-21

51.3% · raw 51.3 %

Headline · max effort · Own vendor

Source/record date: 2026-09-30

Private set via official Zapier public leaderboard; ranks #1 | Highest thinking settings per Google eval methodology.

49.6% · raw 49.6 %

Headline · max effort · Own vendor

Source/record date: 2026-09-02

from chart/figure Meta Muse Spark 1.3 scorecard; AutomationBench E2E; max effort

48.8% · raw 48.8 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-09-10

As reported in DeepSeek-V4.1-Flash HF comparison table (GLM-5.3 column)

48.2% · raw 48.2 %

Alternative · unknown effort · Peer vendor

Source/record date: 2026-09-08

As reported in Nex-N2.5-Pro model card comparison table

48.8% · raw 48.8 %

Headline · unknown effort · Own vendor

Source/record date: 2026-08-26

44.7% · raw 44.7 %

Headline · max effort · Own vendor

Source/record date: 2026-09-28

Same Zapier 1.0.6 run, mirrored per project convention

44.2% · raw 44.2 %

Headline · unknown effort · Own vendor

Source/record date: 2026-09-08

AutomationBench Public v1.0.6

41.4% · raw 41.4 %

Headline · unknown effort · Own vendor

Source/record date: 2026-09-03

41.4% · raw 41.4 %

Alternative · unknown effort · Peer vendor

Source/record date: 2026-09-30

As reported by Google Gemini 4 Argon chart; peer harness notes in Google eval methodology | Same value as current headline; kept as corroboration, non-headline.

40% · raw 40 %

Headline · max effort · Own vendor

Source/record date: 2026-09-22

Anthropic announce table; Zapier AutomationBench without fallback; adaptive thinking max effort

42.5% · raw 42.5 %

Alternative · unknown effort · Peer vendor

Source/record date: 2026-09-30

As reported by Google Gemini 4 Argon chart; peer harness notes in Google eval methodology | Differs from current headline 40.0 (first-party); kept with provenance, non-headline.

39.8% · raw 39.8 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-09-08

As reported in Nex-N2.5-Pro model card comparison table

36.1% · raw 36.1 %

Headline · max effort · Own vendor

Source/record date: 2026-09-29

Max effort; +2.9 over GPT-6 Sol; -5.3 vs Astra 41.4; -6.4 vs Opus 5.5 with fallbacks 42.5

31.7% · raw 31.7 %

Alternative · medium effort · Own vendor

Source/record date: 2026-09-29

Medium effort: +2.2 over Opus 5.5 (29.5), +4.8 over GPT-6 Sol same setting

32.1% · raw 32.1 %

Headline · unknown effort · Own vendor

Source/record date: 2026-08-28

Vendor chart transcription

31.4% · raw 31.4 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-09-03

As reported by OpenAI

31.4% · raw 31.4 %

Alternative · unknown effort · Peer vendor

Source/record date: 2026-09-30

As reported by Google Gemini 4 Argon chart; peer harness notes in Google eval methodology | Same value as current headline; kept as corroboration, non-headline.

30.8% · raw 30.8 %

Headline · max effort · Own vendor

Source/record date: 2026-07-16

AutomationBench 600-task public subset (mapped to automationbench-v1.0.6)

30.3% · raw 30.3 %

Headline · unknown effort · Own vendor

Source/record date: 2026-09-21

Released SFT checkpoint, avg@1; MiMo-V2.6 tech report Table 6 (https://huggingface.co/XiaomiMiMo/MiMo-V2.6-Pro-RL/blob/main/MiMo_V2_6_technical_report.pdf) and HF card Evaluation table. Also plotted (SFT line) in radar figure on https://mimo.mi.com/docs/en-US/news/latest/v2-6. Eval effort not specified (thinking model).