Examenos

50.3% · raw 50.3 %

Headline · max effort · Peer vendor

Source/record date: 2026-09-20

Harness: StepFun vendor-reported evaluation; Banking domain, not tau2 or telecom. Full agent/grader/task-subset settings unpublished unless stated. Source inspected 8 October 2026; evaluation date unknown unless stated. Approved peer, max effort as labelled; not an independent run.

50% · raw 50 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-03

Harness: Aleph Alpha eval-framework/Harbor, released post-trained checkpoint. Kolibri reasoning effort high; peers use vendor-documented sampling. Technical report Tables 27-29 (pp. 96-101). Banking alltools retrieval configuration; avg@4; context overflow scores zero. As reported by Aleph Alpha for Qwen3.8 27B; preserves existing own-vendor headline when present.

47.2% · raw 47.2 %

Headline · max effort · Peer vendor

Source/record date: 2026-09-20

Harness: StepFun vendor-reported evaluation; Banking domain, not tau2 or telecom. Full agent/grader/task-subset settings unpublished unless stated. Source inspected 8 October 2026; evaluation date unknown unless stated. Approved peer, max effort as labelled; not an independent run.

42.5% · raw 42.5 %

Headline · high effort · Own vendor

Source/record date: 2026-09-20

Harness: StepFun vendor-reported evaluation; Banking domain, not tau2 or telecom. Full agent/grader/task-subset settings unpublished unless stated. Source inspected 8 October 2026; evaluation date unknown unless stated. Step 5 Preview high effort.

41.4% · raw 41.4 %

Headline · max effort · Peer vendor

Source/record date: 2026-09-20

Harness: StepFun vendor-reported evaluation; Banking domain, not tau2 or telecom. Full agent/grader/task-subset settings unpublished unless stated. Source inspected 8 October 2026; evaluation date unknown unless stated. Approved peer, max effort as labelled; not an independent run.

38.1% · raw 38.1 %

Headline · high effort · Own vendor

Source/record date: 2026-10-03

Harness: Aleph Alpha eval-framework/Harbor, released post-trained checkpoint. Kolibri reasoning effort high; peers use vendor-documented sampling. Technical report Tables 27-29 (pp. 96-101). Banking alltools retrieval configuration; avg@4; context overflow scores zero.

36.1% · raw 36.1 %

Headline · unknown effort · Own vendor

Source/record date: 2026-10-17

Harness: Reflection final instruction-tuned BF16; temp=0.7, top_p=0.99. Effort and run date unpublished. Earlier preview results are retained separately.

36.8% · raw 36.8 %

Alternative · unknown effort · Own vendor

Source/record date: 2026-10-17

Harness: Reflection final instruction-tuned FP8; temp=0.7, top_p=0.99. Effort and run date unpublished. Earlier preview results are retained separately.

35.4% · raw 35.4 %

Alternative · unknown effort · Own vendor

Source/record date: 2026-10-17

Harness: Reflection final instruction-tuned NVFP4; temp=0.7, top_p=0.99. Effort and run date unpublished. Earlier preview results are retained separately.

38% · raw 38 %

Alternative · unknown effort · Own vendor

Source/record date: 2026-10-05

Harness: Reflection launch evaluation; full agent, revision and token-budget settings not published. Beam early-access preview, reported by Reflection. Demoted 2026-10-17: retained preview observation; released BF16 card now supplies the current headline.

34.9% · raw 34.85 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-15

Harness: As reported by NVIDIA for the exact approved peer. NeMo Gym; report Table 10 and §§3.8–3.9. AA Omniscience uses 600 public questions / 10 repeats with GPT-5.6 Luna judge. SciCode uses scientist background / 8 repeats and distinguishes subtask vs whole-problem accuracy. GPQA averages 8 repeats; APEX Shortlist 16 repeats, no tools. HLE text-only has 2,158 questions; full HLE 2,500 with one sample. Video-MME v2 has no subtitles; V* has no search tool. OSWorld excludes Google Drive tasks (361), screenshot-only pyautogui, 100 steps and partial-credit mean over 3 runs. GraphWalks averages F1 across BFS and parents on the same 950/1,150 prompts fitting ≤1M. IFBench is mean fraction of constraints satisfied under strict verification, not prompt all-pass. Checked 15 October 2026; execution date and selectable model effort unpublished unless explicitly stated.

34.8% · raw 34.8 %

Alternative · unknown effort · Peer vendor

Source/record date: 2026-10-15

Harness: As reported by NVIDIA for the exact approved peer. Exact released BF16 card table. Conflicts/rounding versus detailed report remain non-headline alternatives; detailed report supplies the precise protocol. Raw card label: tau3-bench banking. Checked 15 October 2026; execution date and selectable model effort unpublished unless explicitly stated. Alternative source/configuration; does not replace the existing headline.

33.4% · raw 33.4 %

Headline · max effort · Own vendor

Source/record date: 2026-07-16

τ³-Banking; Moonshot cites Artificial Analysis as of 2026-07-23 Provenance corrected 2026-10-06: published on Moonshot’s own model card; vendor quotes AA, not a peer-vendor comparison.

37.1% · raw 37.1 %

Alternative · unknown effort · Peer vendor

Source/record date: 2026-10-05

Harness: Reflection launch evaluation; full agent, revision and token-budget settings not published. As reported by Reflection; peer comparison, not an independent evaluator record.

46% · raw 46 %

Alternative · max effort · Peer vendor

Source/record date: 2026-09-20

Harness: StepFun vendor-reported evaluation; Banking domain, not tau2 or telecom. Full agent/grader/task-subset settings unpublished unless stated. Source inspected 8 October 2026; evaluation date unknown unless stated. Approved peer, max effort as labelled; not an independent run. Existing headline retained; alternate vendor configuration/evidence, no averaging.

30.2% · raw 30.24 %

Headline · medium effort · Own vendor

Source/record date: 2026-10-19

Inception launch comparison; exact reported value 30.24. Own-model vendor measurement. Banking domain; simulator, harness and repeat aggregation are unpublished. All three named figure columns use medium. Execution date, seeds, task budgets and detailed harness are unpublished unless stated. Vendor claims are separate from evaluator-run results.

23% · raw 23 %

Headline · medium effort · Peer vendor

Source/record date: 2026-10-19

Inception launch comparison; exact reported value 23. Peer value as reported by Inception; existing own-vendor headline retained. Banking domain; simulator, harness and repeat aggregation are unpublished. All three named figure columns use medium. Execution date, seeds, task budgets and detailed harness are unpublished unless stated. Vendor claims are separate from evaluator-run results.

23% · raw 23 %

Headline · unknown effort · Own vendor

Source/record date: 2026-08-11

Harness: Upstage vendor-cited Artificial Analysis August 2026 observation, not direct current AA data. The chart labels these internally measured while prose cites AA; provenance disagreement is retained and not resolved by guessing. Exact approved Solar Pro 4; Solar Open 2 is a separate, unapproved comparison peer. Checked 16 October 2026.

18.9% · raw 18.88 %

Headline · unknown effort · Own vendor

Source/record date: 2026-10-15

Harness: NVIDIA evaluation of the released BF16 post-trained model. NeMo Gym; report Table 10 and §§3.8–3.9. AA Omniscience uses 600 public questions / 10 repeats with GPT-5.6 Luna judge. SciCode uses scientist background / 8 repeats and distinguishes subtask vs whole-problem accuracy. GPQA averages 8 repeats; APEX Shortlist 16 repeats, no tools. HLE text-only has 2,158 questions; full HLE 2,500 with one sample. Video-MME v2 has no subtitles; V* has no search tool. OSWorld excludes Google Drive tasks (361), screenshot-only pyautogui, 100 steps and partial-credit mean over 3 runs. GraphWalks averages F1 across BFS and parents on the same 950/1,150 prompts fitting ≤1M. IFBench is mean fraction of constraints satisfied under strict verification, not prompt all-pass. Checked 15 October 2026; execution date and selectable model effort unpublished unless explicitly stated.

19.9% · raw 19.85 %

Alternative · unknown effort · Own vendor

Source/record date: 2026-10-15

Harness: NVIDIA evaluation of the released BF16 post-trained model. Bolt report Table 7: explicit BF16 columns; all other metrics inherit Super parent evaluation setup. GDPVal is a published index, not Elo or percent; its mapping to Table 10 normalization is not asserted. MCP-Atlas uses the official evaluator on 500 public tasks with reasoning on, not the full 1,000-task suite. WebArena uses 812 public tasks, screenshot vision, vLLM temp 1.0/top_p 0.95, 100 steps, internal string/URL/programmatic checks. BrowseComp uses Exa. Super values differ from its main launch run and remain alternatives. Checked 15 October 2026; execution date and selectable model effort unpublished unless explicitly stated. Alternative source/configuration; does not replace the existing headline.

17% · raw 17.01 %

Headline · unknown effort · Own vendor

Source/record date: 2026-10-15

Harness: NVIDIA evaluation of the released BF16 post-trained model. Bolt report Table 7: explicit BF16 columns; all other metrics inherit Super parent evaluation setup. GDPVal is a published index, not Elo or percent; its mapping to Table 10 normalization is not asserted. MCP-Atlas uses the official evaluator on 500 public tasks with reasoning on, not the full 1,000-task suite. WebArena uses 812 public tasks, screenshot vision, vLLM temp 1.0/top_p 0.95, 100 steps, internal string/URL/programmatic checks. BrowseComp uses Exa. Super values differ from its main launch run and remain alternatives. Checked 15 October 2026; execution date and selectable model effort unpublished unless explicitly stated.

14% · raw 14.02 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-19

Harness: NVIDIA same-suite vendor comparison of exact Gemma4 26B-A4B-it (confirmed figure label). As reported by NVIDIA; effort and inference precision unpublished for the peer.

13.1% · raw 13.06 %

Headline · medium effort · Peer vendor

Source/record date: 2026-10-19

Inception launch comparison; exact reported value 13.06. Peer value as reported by Inception; existing own-vendor headline retained. Banking domain; simulator, harness and repeat aggregation are unpublished. All three named figure columns use medium. Execution date, seeds, task budgets and detailed harness are unpublished unless stated. Vendor claims are separate from evaluator-run results.

9.3% · raw 9.28 %

Headline · unknown effort · Own vendor

Source/record date: 2026-10-19

Harness: NVIDIA release NeMo Gym/Evaluator; BF16 reference weights. τ³-bench (Banking). Source comparison metrics are vendor-run, not independent AA. Thinking enabled without an API effort label; benchmark-specific sampling/repeats are in the official recipes.

8% · raw 8.04 %

Headline · unknown effort · Own vendor

Source/record date: 2026-10-13

Harness: Liquid AI vendor evaluation of the released reasoning-only checkpoint. Official GPT-5.2 user model at low reasoning effort, default BM25 retrieval; agent temperature 0, maximum output 16,384 tokens per turn. The user simulator effort is not the evaluated model effort. Evaluation date unpublished; source checked 13 October 2026. No selectable effort level is specified.