Examenos

95.4% · raw 95.4 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-11-05

Thinking Machines’ Introducing Inkling 2 Small (model card, 5 November 2026), benchmark table, row “SWEBench Verified”, column “GLM 5.3”, value “95.4%”. As reported by Thinking Machines; not a result from the model’s developer. Run attribution for peer columns is not stated. Peer effort and run dates are not published, so effort is unknown.

93.4% · raw 93.4 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-11-05

Thinking Machines’ Introducing Inkling 2 Small (model card, 5 November 2026), benchmark table, row “SWEBench Verified”, column “Kimi K3”, value “93.4%”. As reported by Thinking Machines; not a result from the model’s developer. Run attribution for peer columns is not stated. Peer effort and run dates are not published, so effort is unknown.

92% · raw 92 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-11-05

Thinking Machines’ Introducing Inkling 2 Small (model card, 5 November 2026), benchmark table, row “SWEBench Verified”, column “GLM-5.3-Flash”, value “92.0%”. As reported by Thinking Machines; not a result from the model’s developer. Run attribution for peer columns is not stated. Peer effort and run dates are not published, so effort is unknown.

88.9% · raw 88.9 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-11-05

Thinking Machines’ Introducing Inkling 2 Small (model card, 5 November 2026), benchmark table, row “SWEBench Verified”, column “Claude Haiku 5.5”, value “88.9%”. As reported by Thinking Machines; not a result from the model’s developer. Run attribution for peer columns is not stated. Peer effort and run dates are not published, so effort is unknown.

84.6% · raw 84.6 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-11-05

Thinking Machines’ Introducing Inkling 2 Small (model card, 5 November 2026), benchmark table, row “SWEBench Verified”, column “DeepSeek-V4.1-Flash-1017”, value “84.6%”. As reported by Thinking Machines; not a result from the model’s developer. Run attribution for peer columns is not stated. Peer effort and run dates are not published, so effort is unknown.

82.2% · raw 82.2 %

Headline · xhigh effort · Own vendor

Source/record date: 2026-11-05

Thinking Machines’ Introducing Inkling 2 Small (model card, 5 November 2026), benchmark table, row “SWEBench Verified”, column “Inkling 2 Small”, value “82.2%”. Harness unpublished. Thinking Machines runs its evaluations at effort 0.99, which the post labels “xhigh”, temperature 1.0 and a 1M-token maximum trajectory (256K for Inkling-Small); stored as xhigh because the vendor names it so. Execution date unpublished; the table calls itself an evaluation snapshot.

81.6% · raw 81.6 %

Headline · unknown effort · Own vendor

Source/record date: 2026-10-17

Harness: Reflection final instruction-tuned BF16; temp=0.7, top_p=0.99. Effort and run date unpublished. Earlier preview results are retained separately.

80.2% · raw 80.2 %

Alternative · unknown effort · Own vendor

Source/record date: 2026-10-17

Harness: Reflection final instruction-tuned FP8; temp=0.7, top_p=0.99. Effort and run date unpublished. Earlier preview results are retained separately.

80.7% · raw 80.7 %

Alternative · unknown effort · Own vendor

Source/record date: 2026-10-17

Harness: Reflection final instruction-tuned NVFP4; temp=0.7, top_p=0.99. Effort and run date unpublished. Earlier preview results are retained separately.

80.9% · raw 80.9 %

Alternative · unknown effort · Own vendor

Source/record date: 2026-10-05

Harness: Reflection launch evaluation; full agent, revision and token-budget settings not published. Beam early-access preview, reported by Reflection. Demoted 2026-10-17: retained preview observation; released BF16 card now supplies the current headline.

81.1% · raw 81.1 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-11-05

Thinking Machines’ Introducing Inkling 2 Small (model card, 5 November 2026), benchmark table, row “SWEBench Verified”, column “MiniMax-M3.1-Flash”, value “81.1%”. As reported by Thinking Machines; not a result from the model’s developer. Run attribution for peer columns is not stated. Peer effort and run dates are not published, so effort is unknown.

80.2% · raw 80.2 %

Headline · xhigh effort · Own vendor

Source/record date: 2026-07-30

Thinking Machines’ Inkling-Small post (30 July 2026), main benchmark table, row “SWEBench Verified”, column “Inkling-Small”; the Hugging Face card (revision 8cc5877b44d3) repeats the table exactly. Bash-only harness for Thinking Machines’ models. Thinking Machines runs its own models at effort 0.99 (a 0–1 setting) and temperature 1.0; Effort 0.99 is stored as xhigh: Thinking Machines’ Inkling 2 Small post (5 November 2026, https://thinkingmachines.ai/news/inkling-2-small/) names this setting “effort 0.99 (xhigh)”; relabelled 7 November 2026 from unknown. Execution date unpublished.

80% · raw 80 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-11-05

Thinking Machines’ Introducing Inkling 2 Small (model card, 5 November 2026), benchmark table, row “SWEBench Verified”, column “Gemini 3.8 Flash”, value “80.0%”. As reported by Thinking Machines; not a result from the model’s developer. Run attribution for peer columns is not stated. Peer effort and run dates are not published, so effort is unknown.

80% · raw 80 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-11-05

Thinking Machines’ Introducing Inkling 2 Small (model card, 5 November 2026), benchmark table, row “SWEBench Verified”, column “Mistral Large 4”, value “80.0%”. As reported by Thinking Machines; not a result from the model’s developer. Run attribution for peer columns is not stated. The table lists 49B active / 1050B total for this column; Mistral publishes 52B active / 1.05T total. Peer effort and run dates are not published, so effort is unknown.

78.7% · raw 78.7 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-11-04

Ricursive in-house run (italic in the Ripple-65B Preview card), row “SWE-bench Verified”, column “Muse Spark Open 816B”; not a result from the model’s developer. Harness: OpenCode (the card says Ricursive uses OpenCode wherever no harness is named). Ricursive: all models tested at full BF16 precision with maximum effort levels; the peer’s exact effort name is unstated. Non-italic peer values in the table are the developers’ own figures and are not stored.

80% · raw 80 %

Alternative · unknown effort · Peer vendor

Source/record date: 2026-11-05

Thinking Machines’ Introducing Inkling 2 Small (model card, 5 November 2026), benchmark table, row “SWEBench Verified”, column “Muse Spark Open”, value “80.0%”. As reported by Thinking Machines; not a result from the model’s developer. Run attribution for peer columns is not stated. Peer effort and run dates are not published, so effort is unknown.

77.6% · raw 77.6 %

Headline · xhigh effort · Own vendor

Source/record date: 2026-07-15

Thinking Machines’ Introducing Inkling post (15 July 2026), main benchmark table, row “SWEBench Verified”, column “Inkling effort=0.99”. Bash-only harness for Thinking Machines’ models. Thinking Machines runs its own models at effort 0.99 (a 0–1 setting) and temperature 1.0; Effort 0.99 is stored as xhigh: Thinking Machines’ Inkling 2 Small post (5 November 2026, https://thinkingmachines.ai/news/inkling-2-small/) names this setting “effort 0.99 (xhigh)”; relabelled 7 November 2026 from unknown. Execution date unpublished; the HF card says comparison scores were generated 14 July 2026.

76.5% · raw 76.5 %

Headline · unknown effort · Own vendor

Source/record date: 2026-11-04

Ricursive Ripple-65B Preview Hugging Face card (revision 240d5966cbe0), instruct table, row “SWE-bench Verified”. Harness: OpenCode (the card says Ricursive uses OpenCode wherever no harness is named). Ricursive: all models tested at full BF16 precision with maximum effort levels; the Ripple chat template has no effort setting, so effort is unknown.

76.1% · raw 76.1 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-11-05

Thinking Machines’ Introducing Inkling 2 Small (model card, 5 November 2026), benchmark table, row “SWEBench Verified”, column “GPT-6 Luna”, value “76.1%”. As reported by Thinking Machines; not a result from the model’s developer. Run attribution for peer columns is not stated. Peer effort and run dates are not published, so effort is unknown.

73.6% · raw 73.6 %

Headline · unknown effort · Own vendor

Source/record date: 2026-10-28

From the DiffuSpace launch blog general-benchmarks.jpg table (same values as the agentic-benchmarks.jpg chart and the 28 October X post): "SWE-bench Verified". Harness: Codex, each task run three times and averaged. Thinking mode; effort label not published.

72.6% · raw 72.6 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-03

Harness: Aleph Alpha eval-framework/Harbor, released post-trained checkpoint. Kolibri reasoning effort high; peers use vendor-documented sampling. Technical report Tables 27-29 (pp. 96-101). OpenCode agent; at most 250 steps; dedicated deployment. As reported by Aleph Alpha for Qwen3.8 27B; preserves existing own-vendor headline when present.

70.7% · raw 70.7 %

Headline · unknown effort · Own vendor

Source/record date: 2026-06-04

NVIDIA Nemotron 3 Ultra BF16 card (revision 77df655d5e9f), benchmark table, row “SWE-Bench Verified”. Collected with the NeMo Evaluator SDK (NeMo Gym, NeMo Skills and Harbor harnesses); pinned settings in NVIDIA’s reproducibility tutorial (https://github.com/NVIDIA-NeMo/Evaluator/blob/main/examples/nemotron/nemotron-3-ultra/reproducibility.md). NeMo Evaluator v0.3 native SWE-bench run (reproducibility notes); the card does not name the agent. Report Figure 17 per-agent runs: OpenHands 70.3, OpenCode 67.3. The technical report repeats this table as Table 10. Effort levels unpublished; execution date unpublished.

65% · raw 65 %

Alternative · unknown effort · Own vendor

Source/record date: 2026-06-04

NVIDIA technical report Figure 17 (page 65), agent × model matrix for SWE-bench Verified, row “ultra”, agent column “Mini SWE Agent (2.3.0)”: printed cell label read from the figure (crop in research-raw). The matrix’s average column (SWE-bench Verified 60.6, Terminal-Bench 2.1 48.9) is not stored. Different agent from the card’s headline run, so non-headline. Effort unpublished. Non-headline: the first-party row (70.7, https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16) holds this cell.

70.4% · raw 70.4 %

Alternative · unknown effort · Own vendor

Source/record date: 2026-06-04

NVIDIA technical report Figure 17 (page 65), agent × model matrix for SWE-bench Verified, row “ultra”, agent column “Pi (v0.72.1)”: printed cell label read from the figure (crop in research-raw). The matrix’s average column (SWE-bench Verified 60.6, Terminal-Bench 2.1 48.9) is not stored. Different agent from the card’s headline run, so non-headline. Effort unpublished. Non-headline: the first-party row (70.7, https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16) holds this cell.

60.3% · raw 60.3 %

Alternative · unknown effort · Own vendor

Source/record date: 2026-06-04

NVIDIA technical report Figure 17 (page 65), agent × model matrix for SWE-bench Verified, row “ultra”, agent column “Claude (2.1.126)”: printed cell label read from the figure (crop in research-raw). The matrix’s average column (SWE-bench Verified 60.6, Terminal-Bench 2.1 48.9) is not stored. Different agent from the card’s headline run, so non-headline. Effort unpublished. Non-headline: the first-party row (70.7, https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16) holds this cell.

69.9% · raw 69.9 %

Alternative · unknown effort · Own vendor

Source/record date: 2026-06-04

NVIDIA technical report Figure 17 (page 65), agent × model matrix for SWE-bench Verified, row “ultra”, agent column “Hermes (v2026.4.30)”: printed cell label read from the figure (crop in research-raw). The matrix’s average column (SWE-bench Verified 60.6, Terminal-Bench 2.1 48.9) is not stored. Different agent from the card’s headline run, so non-headline. Effort unpublished. Non-headline: the first-party row (70.7, https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16) holds this cell.

21.1% · raw 21.1 %

Alternative · unknown effort · Own vendor

Source/record date: 2026-06-04

NVIDIA technical report Figure 17 (page 65), agent × model matrix for SWE-bench Verified, row “ultra”, agent column “Codex (0.128/0.135)”: printed cell label read from the figure (crop in research-raw). The matrix’s average column (SWE-bench Verified 60.6, Terminal-Bench 2.1 48.9) is not stored. Different agent from the card’s headline run, so non-headline. Effort unpublished. Non-headline: the first-party row (70.7, https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16) holds this cell.

66.4% · raw 66.4 %

Headline · high effort · Own vendor

Source/record date: 2026-10-03

Harness: Aleph Alpha eval-framework/Harbor, released post-trained checkpoint. Kolibri reasoning effort high; peers use vendor-documented sampling. Technical report Tables 27-29 (pp. 96-101). OpenCode agent; at most 250 steps; dedicated deployment.

62.8% · raw 62.8 %

Headline · high effort · Own vendor

Source/record date: 2026-10-20

Harness: SparkLLM own run at thinking high per its card, temperature 1.0, top_p 0.95; exact label “SWE Verified” in the card figure. Self-reported; run date unpublished.

61.1% · raw 61.1 %

Headline · unknown effort · Own vendor

Source/record date: 2026-09-21

Released SFT checkpoint, avg@3; MiMo-V2.6 tech report Table 6 (https://huggingface.co/XiaomiMiMo/MiMo-V2.6-Pro-RL/blob/main/MiMo_V2_6_technical_report.pdf) and HF card Evaluation table. Also plotted (SFT line) in radar figure on https://mimo.mi.com/docs/en-US/news/latest/v2-6. Eval effort not specified (thinking model). HF card label 'SWE Verified'.

61.7% · raw 61.7 %

Alternative · unknown effort · Own vendor

Source/record date: 2026-09-21

Harness: mini-harness1 (training harness). SFT checkpoint, tech report Table 7 (coding performance across agent harnesses; PDF p.36). Not the headline (headline = Table 6 / HF card).

63.9% · raw 63.9 %

Alternative · unknown effort · Own vendor

Source/record date: 2026-09-21

Harness: mini-harness2 (training harness). SFT checkpoint, tech report Table 7 (coding performance across agent harnesses; PDF p.36). Not the headline (headline = Table 6 / HF card).

61.7% · raw 61.7 %

Alternative · unknown effort · Own vendor

Source/record date: 2026-09-21

Harness: mini-harness3 (training harness). SFT checkpoint, tech report Table 7 (coding performance across agent harnesses; PDF p.36). Not the headline (headline = Table 6 / HF card).

63.1% · raw 63.1 %

Alternative · unknown effort · Own vendor

Source/record date: 2026-09-21

Harness: mini-harness4 (training harness). SFT checkpoint, tech report Table 7 (coding performance across agent harnesses; PDF p.36). Not the headline (headline = Table 6 / HF card).

58.5% · raw 58.5 %

Alternative · unknown effort · Own vendor

Source/record date: 2026-09-21

Harness: codex (held-out harness). SFT checkpoint, tech report Table 7 (coding performance across agent harnesses; PDF p.36). Not the headline (headline = Table 6 / HF card).

61.7% · raw 61.7 %

Alternative · unknown effort · Own vendor

Source/record date: 2026-09-21

Harness: claude code (held-out harness). SFT checkpoint, tech report Table 7 (coding performance across agent harnesses; PDF p.36). Not the headline (headline = Table 6 / HF card).

65.3% · raw 65.3 %

Alternative · unknown effort · Own vendor

Source/record date: 2026-09-21

Harness: mini-swe-agent (held-out harness). SFT checkpoint, tech report Table 7 (coding performance across agent harnesses; PDF p.36). Not the headline (headline = Table 6 / HF card).

62.3% · raw 62.3 %

Alternative · unknown effort · Own vendor

Source/record date: 2026-09-21

Harness: Mean (7 harnesses) (unweighted mean). SFT checkpoint, tech report Table 7 (coding performance across agent harnesses; PDF p.36). Not the headline (headline = Table 6 / HF card).

59.8% · raw 59.8 %

Headline · high effort · Own vendor

Source/record date: 2026-10-19

Harness: SparkLLM own run at thinking high per its card, temperature 1.0, top_p 0.95; exact label “SWE Verified” in the card figure. Self-reported; run date unpublished.

57.8% · raw 57.8 %

Headline · medium effort · Peer vendor

Source/record date: 2026-10-28

Harness: DiffuSpace reproduction for this peer, general-benchmarks.jpg table (same values as the agentic-benchmarks.jpg chart and the 28 October X post): "SWE-bench Verified". Harness: Codex, each task run three times and averaged. Mercury 2.5 Thinking Medium per the table note.

57% · raw 57 %

Headline · unknown effort · Own vendor

Source/record date: 2026-08-25

IBM Granite 4.2 30B card (revision 9e668ce1c538), evaluation table row “SWE Bench Verified”, column “30B Dense”. IBM evaluates with a framework based on NVIDIA’s NeMo Evaluator SDK; thinking mode, effort, sampling and repeats for this table are unstated. Agent harness unstated. Execution date unpublished.

47.7% · raw 47.67 %

Headline · unknown effort · Own vendor

Source/record date: 2026-08-25

IBM Granite 4.2 8B card (revision a7042144125c), evaluation table row “SWE Bench Verified”, column “8B Dense”. IBM evaluates with a framework based on NVIDIA’s NeMo Evaluator SDK; thinking mode, effort, sampling and repeats for this table are unstated. Agent harness unstated. Execution date unpublished.

47% · raw 47 %

Headline · unknown effort · Own vendor

Source/record date: 2026-10-08

Harness: JetBrains pipeline, thinking mode; Pi v0.73.1 agent harness (shell and file tools), each model default sampling, 114K-token context, up to 16K tokens per turn. JetBrains self-reported on the released Mellum2.1 Thinking checkpoint; evaluation date unpublished; thinking mode has no effort tier.

45.6% · raw 45.6 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-28

Harness: DiffuSpace in-house run for this peer under the same settings as Clover, on-device-benchmarks.jpg table: "SWE-bench Verified". Harness: Codex, each task run three times and averaged.

23% · raw 23 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-08

Harness: JetBrains pipeline, thinking mode; Pi v0.73.1 agent harness (shell and file tools), each model default sampling, 114K-token context, up to 16K tokens per turn. JetBrains-measured peer in the Mellum2.1 card comparison table, not a Google result.