Qwen3.8 27B Benchmarks, Specifications & Availability

73% · raw 73 %

Headline · unknown effort · Own vendor

Source/record date: 2026-08-14

Terminal Bench 2.1 (Terminus); HF card HTML table

76.8% · raw 76.8 %

Alternative · unknown effort · Peer vendor

Source/record date: 2026-10-03

Harness: Aleph Alpha eval-framework/Harbor, released post-trained checkpoint. Kolibri reasoning effort high; peers use vendor-documented sampling. Technical report Tables 27-29 (pp. 96-101). Terminus 2 agent; max 250 steps; reply cap 80k tokens or half the context; avg@3. As reported by Aleph Alpha for Qwen3.8 27B; preserves existing own-vendor headline when present.

61.7% · raw 61.7 %

Headline · unknown effort · Own vendor

Source/record date: 2026-08-14

SWE-bench Pro; Claude Code harness (temp=1.0, top_p=0.95, 256K) except Opus uses official

42.3% · raw 42.3 %

Headline · unknown effort · Own vendor

Source/record date: 2026-08-14

NL2Repo-Bench; Claude Code harness

42.2% · raw 42.2 %

Headline · unknown effort · Own vendor

Source/record date: 2026-08-14

DeepSWE 1.1; Claude Code harness

79% · raw 79 %

Headline · unknown effort · Own vendor

Source/record date: 2026-08-14

Card lists QwenSWEBench (mapped to qwenswebench-v2); Claude Code avg@3

70.7% · raw 70.7 %

Headline · unknown effort · Own vendor

Source/record date: 2026-08-14

CoWorkBench (in-house)

33.4% · raw 33.4 %

Headline · unknown effort · Own vendor

Source/record date: 2026-08-14

JobBench

20.4% · raw 20.4 %

Headline · unknown effort · Own vendor

Source/record date: 2026-08-14

Agents' Last Exam Pass@1 (Score also reported 42.9 on card)

79.5% · raw 79.5 %

Headline · unknown effort · Own vendor

Source/record date: 2026-08-14

IFBench

89.2% · raw 89.2 %

Headline · unknown effort · Own vendor

Source/record date: 2026-08-14

GPQA Diamond

89.2% · raw 89.2 %

Alternative · unknown effort · Peer vendor

Source/record date: 2026-10-03

Harness: Aleph Alpha eval-framework/Harbor, released post-trained checkpoint. Kolibri reasoning effort high; peers use vendor-documented sampling. Technical report Tables 27-29 (pp. 96-101). Chain-of-thought, avg@8. As reported by Aleph Alpha for Qwen3.8 27B; preserves existing own-vendor headline when present.

30.8% · raw 30.8 %

Headline · unknown effort · Own vendor

Source/record date: 2026-08-14

Humanity's Last Exam (text; no tools called out)

90.3% · raw 90.3 %

Headline · unknown effort · Own vendor

Source/record date: 2026-08-14

LiveCodeBench v6

93.8% · raw 93.8 %

Alternative · unknown effort · Peer vendor

Source/record date: 2026-10-03

Harness: Aleph Alpha eval-framework/Harbor, released post-trained checkpoint. Kolibri reasoning effort high; peers use vendor-documented sampling. Technical report Tables 27-29 (pp. 96-101). August 2024-April 2025 problems; avg@3. As reported by Aleph Alpha for Qwen3.8 27B; preserves existing own-vendor headline when present.

84.3% · raw 84.3 %

Headline · unknown effort · Own vendor

Source/record date: 2026-08-14

OSWorld-Verified; from HF VL table

64.8% · raw 64.8 %

Headline · unknown effort · Own vendor

Source/record date: 2026-08-14

WebArena-Verified under OSWorld scaffold

81.9% · raw 81.9 %

Headline · unknown effort · Own vendor

Source/record date: 2026-08-14

AndroidWorld

47.1% · raw 47.1 %

Headline · unknown effort · Own vendor

Source/record date: 2026-08-14

RecreationBench (in-house application recreation)

57.4% · raw 57.4 %

Headline · unknown effort · Own vendor

Source/record date: 2026-08-14

ClawEval-MM Pass@3 (Average also reported 56.9)

38.6% · raw 38.6 %

Headline · unknown effort · Own vendor

Source/record date: 2026-08-14

SWE-MM; Claude Code harness on public SWE-bench Multimodal dev split

62.9% · raw 62.9 %

Headline · unknown effort · Own vendor

Source/record date: 2026-08-14

Vision2Web avg across frontend/webpage/website; Claude Code harness

94.6% · raw 94.6 %

Headline · unknown effort · Own vendor

Source/record date: 2026-08-14

MathVision With CI (Without CI 90.0 also on card)

85.6% · raw 85.6 %

Headline · unknown effort · Own vendor

Source/record date: 2026-08-14

BabyVision With CI (Without CI 65.7 also on card)

90.2% · raw 90.2 %

Headline · unknown effort · Own vendor

Source/record date: 2026-08-14

CharXiv (RQ) With CI (Without CI 83.7 also on card)

91.1% · raw 91.1 %

Headline · unknown effort · Own vendor

Source/record date: 2026-08-14

OmniDocBench 1.5

85.9% · raw 85.9 %

Headline · unknown effort · Own vendor

Source/record date: 2026-08-14

RealWorldQA

65.5% · raw 65.5 %

Headline · unknown effort · Own vendor

Source/record date: 2026-08-14

ERQA

80.2% · raw 80.2 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-03

Harness: Aleph Alpha eval-framework/Harbor, released post-trained checkpoint. Kolibri reasoning effort high; peers use vendor-documented sampling. Technical report Tables 27-29 (pp. 96-101). Vendor aggregate excludes incomplete/non-percentage rows; Overall EN uses eight categories, DE four. Long Context excluded. As reported by Aleph Alpha for Qwen3.8 27B; preserves existing own-vendor headline when present.

79.9% · raw 79.9 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-03

Harness: Aleph Alpha eval-framework/Harbor, released post-trained checkpoint. Kolibri reasoning effort high; peers use vendor-documented sampling. Technical report Tables 27-29 (pp. 96-101). German translation (ellamind), except MMLU-ProX and vendor retrieval/proxy tasks. Vendor aggregate excludes incomplete/non-percentage rows; Overall EN uses eight categories, DE four. Long Context excluded. As reported by Aleph Alpha for Qwen3.8 27B; preserves existing own-vendor headline when present.

56.8% · raw 56.8 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-03

Harness: Aleph Alpha eval-framework/Harbor, released post-trained checkpoint. Kolibri reasoning effort high; peers use vendor-documented sampling. Technical report Tables 27-29 (pp. 96-101). Vendor aggregate excludes incomplete/non-percentage rows; Overall EN uses eight categories, DE four. Long Context excluded. As reported by Aleph Alpha for Qwen3.8 27B; preserves existing own-vendor headline when present.

69.2% · raw 69.2 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-03

Harness: Aleph Alpha eval-framework/Harbor, released post-trained checkpoint. Kolibri reasoning effort high; peers use vendor-documented sampling. Technical report Tables 27-29 (pp. 96-101). German translation (ellamind), except MMLU-ProX and vendor retrieval/proxy tasks. Vendor aggregate excludes incomplete/non-percentage rows; Overall EN uses eight categories, DE four. Long Context excluded. As reported by Aleph Alpha for Qwen3.8 27B; preserves existing own-vendor headline when present.

88.1% · raw 88.1 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-03

Harness: Aleph Alpha eval-framework/Harbor, released post-trained checkpoint. Kolibri reasoning effort high; peers use vendor-documented sampling. Technical report Tables 27-29 (pp. 96-101). German translation (ellamind), except MMLU-ProX and vendor retrieval/proxy tasks. As reported by Aleph Alpha for Qwen3.8 27B; preserves existing own-vendor headline when present.

35.6% · raw 35.6 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-03

Harness: Aleph Alpha eval-framework/Harbor, released post-trained checkpoint. Kolibri reasoning effort high; peers use vendor-documented sampling. Technical report Tables 27-29 (pp. 96-101). Text-only questions, no output cap. As reported by Aleph Alpha for Qwen3.8 27B; preserves existing own-vendor headline when present.

37.2% · raw 37.2 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-03

Harness: Aleph Alpha eval-framework/Harbor, released post-trained checkpoint. Kolibri reasoning effort high; peers use vendor-documented sampling. Technical report Tables 27-29 (pp. 96-101). German translation (ellamind), except MMLU-ProX and vendor retrieval/proxy tasks. As reported by Aleph Alpha for Qwen3.8 27B; preserves existing own-vendor headline when present.

19% · raw 19 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-03

Harness: Aleph Alpha eval-framework/Harbor, released post-trained checkpoint. Kolibri reasoning effort high; peers use vendor-documented sampling. Technical report Tables 27-29 (pp. 96-101). As reported by Aleph Alpha for Qwen3.8 27B; preserves existing own-vendor headline when present. Source conflict: report gives 17.5; card gives 19.0. Card is the featured source; report retained as alternative.

17.5% · raw 17.5 %

Alternative · unknown effort · Peer vendor

Source/record date: 2026-10-03

Harness: Aleph Alpha eval-framework/Harbor, released post-trained checkpoint. Kolibri reasoning effort high; peers use vendor-documented sampling. Technical report Tables 27-29 (pp. 96-101). As reported by Aleph Alpha for Qwen3.8 27B; preserves existing own-vendor headline when present. Source conflict: report gives 17.5; card gives 19.0. Card is the featured source; report retained as alternative.

-5.8 · raw -5.8 index

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-03

Harness: Aleph Alpha eval-framework/Harbor, released post-trained checkpoint. Kolibri reasoning effort high; peers use vendor-documented sampling. Technical report Tables 27-29 (pp. 96-101). As reported by Aleph Alpha for Qwen3.8 27B; preserves existing own-vendor headline when present. Source conflict: report gives -9.5; card gives -5.8. Card is the featured source; report retained as alternative.

-9.5 · raw -9.5 index

Alternative · unknown effort · Peer vendor

Source/record date: 2026-10-03

Harness: Aleph Alpha eval-framework/Harbor, released post-trained checkpoint. Kolibri reasoning effort high; peers use vendor-documented sampling. Technical report Tables 27-29 (pp. 96-101). As reported by Aleph Alpha for Qwen3.8 27B; preserves existing own-vendor headline when present. Source conflict: report gives -9.5; card gives -5.8. Card is the featured source; report retained as alternative.

85% · raw 85 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-03

Harness: Aleph Alpha eval-framework/Harbor, released post-trained checkpoint. Kolibri reasoning effort high; peers use vendor-documented sampling. Technical report Tables 27-29 (pp. 96-101). As reported by Aleph Alpha for Qwen3.8 27B; preserves existing own-vendor headline when present.

82.4% · raw 82.4 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-03

Harness: Aleph Alpha eval-framework/Harbor, released post-trained checkpoint. Kolibri reasoning effort high; peers use vendor-documented sampling. Technical report Tables 27-29 (pp. 96-101). German translation (ellamind), except MMLU-ProX and vendor retrieval/proxy tasks. As reported by Aleph Alpha for Qwen3.8 27B; preserves existing own-vendor headline when present.

97.8% · raw 97.8 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-03

Harness: Aleph Alpha eval-framework/Harbor, released post-trained checkpoint. Kolibri reasoning effort high; peers use vendor-documented sampling. Technical report Tables 27-29 (pp. 96-101). Vendor aggregate excludes incomplete/non-percentage rows; Overall EN uses eight categories, DE four. Long Context excluded. As reported by Aleph Alpha for Qwen3.8 27B; preserves existing own-vendor headline when present.

96.7% · raw 96.7 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-03

Harness: Aleph Alpha eval-framework/Harbor, released post-trained checkpoint. Kolibri reasoning effort high; peers use vendor-documented sampling. Technical report Tables 27-29 (pp. 96-101). German translation (ellamind), except MMLU-ProX and vendor retrieval/proxy tasks. Vendor aggregate excludes incomplete/non-percentage rows; Overall EN uses eight categories, DE four. Long Context excluded. As reported by Aleph Alpha for Qwen3.8 27B; preserves existing own-vendor headline when present.

97.9% · raw 97.9 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-03

Harness: Aleph Alpha eval-framework/Harbor, released post-trained checkpoint. Kolibri reasoning effort high; peers use vendor-documented sampling. Technical report Tables 27-29 (pp. 96-101). avg@16. As reported by Aleph Alpha for Qwen3.8 27B; preserves existing own-vendor headline when present.

96.5% · raw 96.5 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-03

Harness: Aleph Alpha eval-framework/Harbor, released post-trained checkpoint. Kolibri reasoning effort high; peers use vendor-documented sampling. Technical report Tables 27-29 (pp. 96-101). German translation (ellamind), except MMLU-ProX and vendor retrieval/proxy tasks. avg@16. As reported by Aleph Alpha for Qwen3.8 27B; preserves existing own-vendor headline when present.

97.7% · raw 97.7 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-03

Harness: Aleph Alpha eval-framework/Harbor, released post-trained checkpoint. Kolibri reasoning effort high; peers use vendor-documented sampling. Technical report Tables 27-29 (pp. 96-101). avg@16. As reported by Aleph Alpha for Qwen3.8 27B; preserves existing own-vendor headline when present.

96.9% · raw 96.9 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-03

Harness: Aleph Alpha eval-framework/Harbor, released post-trained checkpoint. Kolibri reasoning effort high; peers use vendor-documented sampling. Technical report Tables 27-29 (pp. 96-101). German translation (ellamind), except MMLU-ProX and vendor retrieval/proxy tasks. avg@16. As reported by Aleph Alpha for Qwen3.8 27B; preserves existing own-vendor headline when present.

66.7% · raw 66.7 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-03

Harness: Aleph Alpha eval-framework/Harbor, released post-trained checkpoint. Kolibri reasoning effort high; peers use vendor-documented sampling. Technical report Tables 27-29 (pp. 96-101). Vendor aggregate excludes incomplete/non-percentage rows; Overall EN uses eight categories, DE four. Long Context excluded. As reported by Aleph Alpha for Qwen3.8 27B; preserves existing own-vendor headline when present.

82.5% · raw 82.5 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-03

Harness: Aleph Alpha eval-framework/Harbor, released post-trained checkpoint. Kolibri reasoning effort high; peers use vendor-documented sampling. Technical report Tables 27-29 (pp. 96-101). Domain score, avg@3; not the three-domain aggregate. As reported by Aleph Alpha for Qwen3.8 27B; preserves existing own-vendor headline when present.

68.7% · raw 68.7 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-03

Harness: Aleph Alpha eval-framework/Harbor, released post-trained checkpoint. Kolibri reasoning effort high; peers use vendor-documented sampling. Technical report Tables 27-29 (pp. 96-101). Domain score, avg@3; not the three-domain aggregate. As reported by Aleph Alpha for Qwen3.8 27B; preserves existing own-vendor headline when present.

83.3% · raw 83.3 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-03

Harness: Aleph Alpha eval-framework/Harbor, released post-trained checkpoint. Kolibri reasoning effort high; peers use vendor-documented sampling. Technical report Tables 27-29 (pp. 96-101). Domain score, avg@3; not the three-domain aggregate. As reported by Aleph Alpha for Qwen3.8 27B; preserves existing own-vendor headline when present.

50% · raw 50 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-03

Harness: Aleph Alpha eval-framework/Harbor, released post-trained checkpoint. Kolibri reasoning effort high; peers use vendor-documented sampling. Technical report Tables 27-29 (pp. 96-101). Banking alltools retrieval configuration; avg@4; context overflow scores zero. As reported by Aleph Alpha for Qwen3.8 27B; preserves existing own-vendor headline when present.

42.5% · raw 42.5 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-03

Harness: Aleph Alpha eval-framework/Harbor, released post-trained checkpoint. Kolibri reasoning effort high; peers use vendor-documented sampling. Technical report Tables 27-29 (pp. 96-101). As reported by Aleph Alpha for Qwen3.8 27B; preserves existing own-vendor headline when present.

73.2% · raw 73.2 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-03

Harness: Aleph Alpha eval-framework/Harbor, released post-trained checkpoint. Kolibri reasoning effort high; peers use vendor-documented sampling. Technical report Tables 27-29 (pp. 96-101). bfcl-eval; different web-search backend and per-model sampling; not comparable to BFCL leaderboard. Overall includes irrelevance detection not separately printed. As reported by Aleph Alpha for Qwen3.8 27B; preserves existing own-vendor headline when present.

85.3% · raw 85.3 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-03

Harness: Aleph Alpha eval-framework/Harbor, released post-trained checkpoint. Kolibri reasoning effort high; peers use vendor-documented sampling. Technical report Tables 27-29 (pp. 96-101). bfcl-eval; different web-search backend and per-model sampling; not comparable to BFCL leaderboard. Overall includes irrelevance detection not separately printed. As reported by Aleph Alpha for Qwen3.8 27B; preserves existing own-vendor headline when present.

79.9% · raw 79.9 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-03

Harness: Aleph Alpha eval-framework/Harbor, released post-trained checkpoint. Kolibri reasoning effort high; peers use vendor-documented sampling. Technical report Tables 27-29 (pp. 96-101). bfcl-eval; different web-search backend and per-model sampling; not comparable to BFCL leaderboard. Overall includes irrelevance detection not separately printed. As reported by Aleph Alpha for Qwen3.8 27B; preserves existing own-vendor headline when present.

55.5% · raw 55.5 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-03

Harness: Aleph Alpha eval-framework/Harbor, released post-trained checkpoint. Kolibri reasoning effort high; peers use vendor-documented sampling. Technical report Tables 27-29 (pp. 96-101). bfcl-eval; different web-search backend and per-model sampling; not comparable to BFCL leaderboard. Overall includes irrelevance detection not separately printed. As reported by Aleph Alpha for Qwen3.8 27B; preserves existing own-vendor headline when present.

79.6% · raw 79.6 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-03

Harness: Aleph Alpha eval-framework/Harbor, released post-trained checkpoint. Kolibri reasoning effort high; peers use vendor-documented sampling. Technical report Tables 27-29 (pp. 96-101). bfcl-eval; different web-search backend and per-model sampling; not comparable to BFCL leaderboard. Overall includes irrelevance detection not separately printed. As reported by Aleph Alpha for Qwen3.8 27B; preserves existing own-vendor headline when present.

82% · raw 82 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-03

Harness: Aleph Alpha eval-framework/Harbor, released post-trained checkpoint. Kolibri reasoning effort high; peers use vendor-documented sampling. Technical report Tables 27-29 (pp. 96-101). bfcl-eval; different web-search backend and per-model sampling; not comparable to BFCL leaderboard. Overall includes irrelevance detection not separately printed. As reported by Aleph Alpha for Qwen3.8 27B; preserves existing own-vendor headline when present.

46.4% · raw 46.4 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-03

Harness: Aleph Alpha eval-framework/Harbor, released post-trained checkpoint. Kolibri reasoning effort high; peers use vendor-documented sampling. Technical report Tables 27-29 (pp. 96-101). As reported by Aleph Alpha for Qwen3.8 27B; preserves existing own-vendor headline when present.

94.2% · raw 94.2 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-03

Harness: Aleph Alpha eval-framework/Harbor, released post-trained checkpoint. Kolibri reasoning effort high; peers use vendor-documented sampling. Technical report Tables 27-29 (pp. 96-101). Vendor aggregate excludes incomplete/non-percentage rows; Overall EN uses eight categories, DE four. Long Context excluded. As reported by Aleph Alpha for Qwen3.8 27B; preserves existing own-vendor headline when present.

94.7% · raw 94.7 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-03

Harness: Aleph Alpha eval-framework/Harbor, released post-trained checkpoint. Kolibri reasoning effort high; peers use vendor-documented sampling. Technical report Tables 27-29 (pp. 96-101). As reported by Aleph Alpha for Qwen3.8 27B; preserves existing own-vendor headline when present.

72.6% · raw 72.6 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-03

Harness: Aleph Alpha eval-framework/Harbor, released post-trained checkpoint. Kolibri reasoning effort high; peers use vendor-documented sampling. Technical report Tables 27-29 (pp. 96-101). OpenCode agent; at most 250 steps; dedicated deployment. As reported by Aleph Alpha for Qwen3.8 27B; preserves existing own-vendor headline when present.

81.9% · raw 81.9 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-03

Harness: Aleph Alpha eval-framework/Harbor, released post-trained checkpoint. Kolibri reasoning effort high; peers use vendor-documented sampling. Technical report Tables 27-29 (pp. 96-101). Vendor aggregate excludes incomplete/non-percentage rows; Overall EN uses eight categories, DE four. Long Context excluded. As reported by Aleph Alpha for Qwen3.8 27B; preserves existing own-vendor headline when present.

81.9% · raw 81.9 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-03

Harness: Aleph Alpha eval-framework/Harbor, released post-trained checkpoint. Kolibri reasoning effort high; peers use vendor-documented sampling. Technical report Tables 27-29 (pp. 96-101). Loose-prompt metric, avg@5; distinct from strict IFBench scoring. As reported by Aleph Alpha for Qwen3.8 27B; preserves existing own-vendor headline when present.

67.5% · raw 67.5 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-03

Harness: Aleph Alpha eval-framework/Harbor, released post-trained checkpoint. Kolibri reasoning effort high; peers use vendor-documented sampling. Technical report Tables 27-29 (pp. 96-101). Vendor aggregate excludes incomplete/non-percentage rows; Overall EN uses eight categories, DE four. Long Context excluded. As reported by Aleph Alpha for Qwen3.8 27B; preserves existing own-vendor headline when present.

8.3% · raw 8.3 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-03

Harness: Aleph Alpha eval-framework/Harbor, released post-trained checkpoint. Kolibri reasoning effort high; peers use vendor-documented sampling. Technical report Tables 27-29 (pp. 96-101). As reported by Aleph Alpha for Qwen3.8 27B; preserves existing own-vendor headline when present.

85.6% · raw 85.6 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-03

Harness: Aleph Alpha eval-framework/Harbor, released post-trained checkpoint. Kolibri reasoning effort high; peers use vendor-documented sampling. Technical report Tables 27-29 (pp. 96-101). As reported by Aleph Alpha for Qwen3.8 27B; preserves existing own-vendor headline when present.

73% · raw 73 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-03

Harness: Aleph Alpha eval-framework/Harbor, released post-trained checkpoint. Kolibri reasoning effort high; peers use vendor-documented sampling. Technical report Tables 27-29 (pp. 96-101). As reported by Aleph Alpha for Qwen3.8 27B; preserves existing own-vendor headline when present.

70.6% · raw 70.6 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-03

Harness: Aleph Alpha eval-framework/Harbor, released post-trained checkpoint. Kolibri reasoning effort high; peers use vendor-documented sampling. Technical report Tables 27-29 (pp. 96-101). As reported by Aleph Alpha for Qwen3.8 27B; preserves existing own-vendor headline when present.

67.3% · raw 67.3 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-03

Harness: Aleph Alpha eval-framework/Harbor, released post-trained checkpoint. Kolibri reasoning effort high; peers use vendor-documented sampling. Technical report Tables 27-29 (pp. 96-101). As reported by Aleph Alpha for Qwen3.8 27B; preserves existing own-vendor headline when present.

53% · raw 53 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-03

Harness: Aleph Alpha eval-framework/Harbor, released post-trained checkpoint. Kolibri reasoning effort high; peers use vendor-documented sampling. Technical report Tables 27-29 (pp. 96-101). As reported by Aleph Alpha for Qwen3.8 27B; preserves existing own-vendor headline when present.

78.6% · raw 78.6 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-03

Harness: Aleph Alpha eval-framework/Harbor, released post-trained checkpoint. Kolibri reasoning effort high; peers use vendor-documented sampling. Technical report Tables 27-29 (pp. 96-101). As reported by Aleph Alpha for Qwen3.8 27B; preserves existing own-vendor headline when present.

83.3% · raw 83.3 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-03

Harness: Aleph Alpha eval-framework/Harbor, released post-trained checkpoint. Kolibri reasoning effort high; peers use vendor-documented sampling. Technical report Tables 27-29 (pp. 96-101). As reported by Aleph Alpha for Qwen3.8 27B; preserves existing own-vendor headline when present.

86.9% · raw 86.9 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-03

Harness: Aleph Alpha eval-framework/Harbor, released post-trained checkpoint. Kolibri reasoning effort high; peers use vendor-documented sampling. Technical report Tables 27-29 (pp. 96-101). As reported by Aleph Alpha for Qwen3.8 27B; preserves existing own-vendor headline when present.

84% · raw 84 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-03

Harness: Aleph Alpha eval-framework/Harbor, released post-trained checkpoint. Kolibri reasoning effort high; peers use vendor-documented sampling. Technical report Tables 27-29 (pp. 96-101). As reported by Aleph Alpha for Qwen3.8 27B; preserves existing own-vendor headline when present.

94.4% · raw 94.4 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-03

Harness: Aleph Alpha eval-framework/Harbor, released post-trained checkpoint. Kolibri reasoning effort high; peers use vendor-documented sampling. Technical report Tables 27-29 (pp. 96-101). As reported by Aleph Alpha for Qwen3.8 27B; preserves existing own-vendor headline when present.

82.5% · raw 82.5 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-03

Harness: Aleph Alpha eval-framework/Harbor, released post-trained checkpoint. Kolibri reasoning effort high; peers use vendor-documented sampling. Technical report Tables 27-29 (pp. 96-101). As reported by Aleph Alpha for Qwen3.8 27B; preserves existing own-vendor headline when present.

83.7% · raw 83.7 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-03

Harness: Aleph Alpha eval-framework/Harbor, released post-trained checkpoint. Kolibri reasoning effort high; peers use vendor-documented sampling. Technical report Tables 27-29 (pp. 96-101). Vendor aggregate excludes incomplete/non-percentage rows; Overall EN uses eight categories, DE four. Long Context excluded. As reported by Aleph Alpha for Qwen3.8 27B; preserves existing own-vendor headline when present.

73.5% · raw 73.5 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-03

Harness: Aleph Alpha eval-framework/Harbor, released post-trained checkpoint. Kolibri reasoning effort high; peers use vendor-documented sampling. Technical report Tables 27-29 (pp. 96-101). German translation (ellamind), except MMLU-ProX and vendor retrieval/proxy tasks. Vendor aggregate excludes incomplete/non-percentage rows; Overall EN uses eight categories, DE four. Long Context excluded. As reported by Aleph Alpha for Qwen3.8 27B; preserves existing own-vendor headline when present.

83.7% · raw 83.7 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-03

Harness: Aleph Alpha eval-framework/Harbor, released post-trained checkpoint. Kolibri reasoning effort high; peers use vendor-documented sampling. Technical report Tables 27-29 (pp. 96-101). As reported by Aleph Alpha for Qwen3.8 27B; preserves existing own-vendor headline when present.

85% · raw 85 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-03

Harness: Aleph Alpha eval-framework/Harbor, released post-trained checkpoint. Kolibri reasoning effort high; peers use vendor-documented sampling. Technical report Tables 27-29 (pp. 96-101). As reported by Aleph Alpha for Qwen3.8 27B; preserves existing own-vendor headline when present.

73.5% · raw 73.5 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-03

Harness: Aleph Alpha eval-framework/Harbor, released post-trained checkpoint. Kolibri reasoning effort high; peers use vendor-documented sampling. Technical report Tables 27-29 (pp. 96-101). German translation (ellamind), except MMLU-ProX and vendor retrieval/proxy tasks. As reported by Aleph Alpha for Qwen3.8 27B; preserves existing own-vendor headline when present.

93.3% · raw 93.3 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-03

Harness: Aleph Alpha eval-framework/Harbor, released post-trained checkpoint. Kolibri reasoning effort high; peers use vendor-documented sampling. Technical report Tables 27-29 (pp. 96-101). Vendor aggregate excludes incomplete/non-percentage rows; Overall EN uses eight categories, DE four. Long Context excluded. As reported by Aleph Alpha for Qwen3.8 27B; preserves existing own-vendor headline when present. Source conflict: report gives 93.2; card gives 93.3. Card is the featured source; report retained as alternative.

93.2% · raw 93.2 %

Alternative · unknown effort · Peer vendor

Source/record date: 2026-10-03

Harness: Aleph Alpha eval-framework/Harbor, released post-trained checkpoint. Kolibri reasoning effort high; peers use vendor-documented sampling. Technical report Tables 27-29 (pp. 96-101). Vendor aggregate excludes incomplete/non-percentage rows; Overall EN uses eight categories, DE four. Long Context excluded. As reported by Aleph Alpha for Qwen3.8 27B; preserves existing own-vendor headline when present. Source conflict: report gives 93.2; card gives 93.3. Card is the featured source; report retained as alternative.

80.2% · raw 80.2 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-03

Harness: Aleph Alpha eval-framework/Harbor, released post-trained checkpoint. Kolibri reasoning effort high; peers use vendor-documented sampling. Technical report Tables 27-29 (pp. 96-101). German translation (ellamind), except MMLU-ProX and vendor retrieval/proxy tasks. Vendor aggregate excludes incomplete/non-percentage rows; Overall EN uses eight categories, DE four. Long Context excluded. As reported by Aleph Alpha for Qwen3.8 27B; preserves existing own-vendor headline when present.

89.2% · raw 89.2 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-03

Harness: Aleph Alpha eval-framework/Harbor, released post-trained checkpoint. Kolibri reasoning effort high; peers use vendor-documented sampling. Technical report Tables 27-29 (pp. 96-101). As reported by Aleph Alpha for Qwen3.8 27B; preserves existing own-vendor headline when present.

89% · raw 89 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-03

Harness: Aleph Alpha eval-framework/Harbor, released post-trained checkpoint. Kolibri reasoning effort high; peers use vendor-documented sampling. Technical report Tables 27-29 (pp. 96-101). As reported by Aleph Alpha for Qwen3.8 27B; preserves existing own-vendor headline when present.

73.8% · raw 73.8 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-03

Harness: Aleph Alpha eval-framework/Harbor, released post-trained checkpoint. Kolibri reasoning effort high; peers use vendor-documented sampling. Technical report Tables 27-29 (pp. 96-101). As reported by Aleph Alpha for Qwen3.8 27B; preserves existing own-vendor headline when present.

97.3% · raw 97.3 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-03

Harness: Aleph Alpha eval-framework/Harbor, released post-trained checkpoint. Kolibri reasoning effort high; peers use vendor-documented sampling. Technical report Tables 27-29 (pp. 96-101). As reported by Aleph Alpha for Qwen3.8 27B; preserves existing own-vendor headline when present.

71.4% · raw 71.4 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-03

Harness: Aleph Alpha eval-framework/Harbor, released post-trained checkpoint. Kolibri reasoning effort high; peers use vendor-documented sampling. Technical report Tables 27-29 (pp. 96-101). As reported by Aleph Alpha for Qwen3.8 27B; preserves existing own-vendor headline when present.

76.9% · raw 76.9 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-03

Harness: Aleph Alpha eval-framework/Harbor, released post-trained checkpoint. Kolibri reasoning effort high; peers use vendor-documented sampling. Technical report Tables 27-29 (pp. 96-101). As reported by Aleph Alpha for Qwen3.8 27B; preserves existing own-vendor headline when present.

81.3% · raw 81.3 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-03

Harness: Aleph Alpha eval-framework/Harbor, released post-trained checkpoint. Kolibri reasoning effort high; peers use vendor-documented sampling. Technical report Tables 27-29 (pp. 96-101). Vendor-run AA-LCR, avg@3; not an independent AA measurement. As reported by Aleph Alpha for Qwen3.8 27B; preserves existing own-vendor headline when present.

34 · raw 34 index

Headline · xhigh effort · Independent evaluator

Source/record date: 2026-10-03

Current AA model leaderboard;

28 · raw 28 index

Alternative · medium effort · Independent evaluator

Source/record date: 2026-10-03

Current AA model leaderboard;

26 · raw 26 index

Alternative · low effort · Independent evaluator

Source/record date: 2026-10-03

Current AA model leaderboard;

20 · raw 20 index

Alternative · none effort · Independent evaluator

Source/record date: 2026-10-03

Current AA model leaderboard;

46 tok/s · raw 46 tok/s

Headline · xhigh effort · Independent evaluator

Source/record date: 2026-10-03

Current AA model leaderboard; Median output tokens/s; leaderboard rounds to whole tokens.

53 tok/s · raw 53 tok/s

Alternative · medium effort · Independent evaluator

Source/record date: 2026-10-03

Current AA model leaderboard; Median output tokens/s; leaderboard rounds to whole tokens.

52 tok/s · raw 52 tok/s

Alternative · low effort · Independent evaluator

Source/record date: 2026-10-03

Current AA model leaderboard; Median output tokens/s; leaderboard rounds to whole tokens.

51 tok/s · raw 51 tok/s

Alternative · none effort · Independent evaluator

Source/record date: 2026-10-03

Current AA model leaderboard; Median output tokens/s; leaderboard rounds to whole tokens.

48.5% · raw 48.48 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-09-23

Accuracy ±1.45; rank noted 32/59 on page extract

15 fixes · raw 15 fixes

Headline · xhigh effort · Independent evaluator

Source/record date: 2026-10-03

Harness: Claude Code / Alibaba API; effort xhigh; 3 runs; evaluation 2026-09-15. Best documented score for this effort in Oct 1 README. Headline: best documented model run.

64.9% · raw 64.85 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-10-03

Refreshed from current Vals leaderboard. Harness: OpenHands. Cost/test $23.46.

87.5% · raw 87.5 %

Headline · xhigh effort · Independent evaluator

Source/record date: 2026-10-03

Semi-Private verified matrix. Published verified configuration. Headline: best verified value; source matrix effort order breaks ties.

68.7% · raw 68.7 %

Alternative · medium effort · Independent evaluator

Source/record date: 2026-10-03

Semi-Private verified matrix. Published verified configuration.

69.2% · raw 69.2 %

Alternative · low effort · Independent evaluator

Source/record date: 2026-10-03

Semi-Private verified matrix. Published verified configuration.

34% · raw 34 %

Alternative · none effort · Independent evaluator

Source/record date: 2026-10-03

Semi-Private verified matrix. Published verified configuration.

42.4% · raw 42.4 %

Headline · xhigh effort · Independent evaluator

Source/record date: 2026-10-03

Semi-Private verified matrix. Published verified configuration. Headline: best verified value; source matrix effort order breaks ties.

13.2% · raw 13.2 %

Alternative · medium effort · Independent evaluator

Source/record date: 2026-10-03

Semi-Private verified matrix. Published verified configuration.

22.8% · raw 22.8 %

Alternative · low effort · Independent evaluator

Source/record date: 2026-10-03

Semi-Private verified matrix. Published verified configuration.

1.5% · raw 1.5 %

Alternative · none effort · Independent evaluator

Source/record date: 2026-10-03

Semi-Private verified matrix. Published verified configuration.

$1.01 · raw 1.01 USD

Headline · xhigh effort · Independent evaluator

Source/record date: 2026-10-03

Current AA model leaderboard;

$1.13 · raw 1.13 USD

Alternative · medium effort · Independent evaluator

Source/record date: 2026-10-03

Current AA model leaderboard;

$1.05 · raw 1.05 USD

Alternative · low effort · Independent evaluator

Source/record date: 2026-10-03

Current AA model leaderboard;

$2.49 · raw 2.49 USD

Alternative · none effort · Independent evaluator

Source/record date: 2026-10-03

Current AA model leaderboard;