Examenos

40 · raw 40 index

Headline · xhigh effort · Own vendor

Source/record date: 2026-07-30

Thinking Machines’ Inkling-Small post (30 July 2026), main benchmark table, row “AA Index v4.1”, column “Inkling-Small”; the Hugging Face card (revision 8cc5877b44d3) repeats the table exactly. Thinking Machines states it uses Artificial Analysis’s reported score for this benchmark, for its own and other models; not an Artificial Analysis record captured by Examenos. Printed with a % sign; stored as the index value. Thinking Machines runs its own models at effort 0.99 (a 0–1 setting) and temperature 1.0; Effort 0.99 is stored as xhigh: Thinking Machines’ Inkling 2 Small post (5 November 2026, https://thinkingmachines.ai/news/inkling-2-small/) names this setting “effort 0.99 (xhigh)”; relabelled 7 November 2026 from unknown. Execution date unpublished.

80.2% · raw 80.2 %

Headline · xhigh effort · Own vendor

Source/record date: 2026-07-30

Thinking Machines’ Inkling-Small post (30 July 2026), main benchmark table, row “SWEBench Verified”, column “Inkling-Small”; the Hugging Face card (revision 8cc5877b44d3) repeats the table exactly. Bash-only harness for Thinking Machines’ models. Thinking Machines runs its own models at effort 0.99 (a 0–1 setting) and temperature 1.0; Effort 0.99 is stored as xhigh: Thinking Machines’ Inkling 2 Small post (5 November 2026, https://thinkingmachines.ai/news/inkling-2-small/) names this setting “effort 0.99 (xhigh)”; relabelled 7 November 2026 from unknown. Execution date unpublished.

55.9% · raw 55.9 %

Headline · xhigh effort · Own vendor

Source/record date: 2026-07-30

Thinking Machines’ Inkling-Small post (30 July 2026), main benchmark table, row “SWEBench Pro Public”, column “Inkling-Small”; the Hugging Face card (revision 8cc5877b44d3) repeats the table exactly. Public set; harness unpublished. Thinking Machines runs its own models at effort 0.99 (a 0–1 setting) and temperature 1.0; Effort 0.99 is stored as xhigh: Thinking Machines’ Inkling 2 Small post (5 November 2026, https://thinkingmachines.ai/news/inkling-2-small/) names this setting “effort 0.99 (xhigh)”; relabelled 7 November 2026 from unknown. Execution date unpublished.

64.7% · raw 64.7 %

Headline · xhigh effort · Own vendor

Source/record date: 2026-07-30

Thinking Machines’ Inkling-Small post (30 July 2026), main benchmark table, row “Terminal Bench 2.1 Best Harness”, column “Inkling-Small”; the Hugging Face card (revision 8cc5877b44d3) repeats the table exactly. Internal coding harness for Thinking Machines’ models; a small number of solutions found contaminated from web search were scored 0. Thinking Machines runs its own models at effort 0.99 (a 0–1 setting) and temperature 1.0; Effort 0.99 is stored as xhigh: Thinking Machines’ Inkling 2 Small post (5 November 2026, https://thinkingmachines.ai/news/inkling-2-small/) names this setting “effort 0.99 (xhigh)”; relabelled 7 November 2026 from unknown. Execution date unpublished.

53.4% · raw 53.4 %

Alternative · unknown effort · Peer vendor

Source/record date: 2026-10-15

As reported by NVIDIA for the exact peer in its Nemotron 3.5 Super and Lightning technical report (15 October 2026), Table 10, column “Inkling Small”, row “Terminal-Bench2.1”; NVIDIA’s run, not a Thinking Machines result. NeMo Gym; report Table 10 and §§3.8–3.9. Terminus 2 with a 3-hour task timeout. Execution date and effort unpublished. Non-headline: the first-party row (64.7, https://thinkingmachines.ai/news/inkling-small/) holds this cell.

48.7% · raw 48.7 %

Headline · xhigh effort · Own vendor

Source/record date: 2026-07-30

Thinking Machines’ Inkling-Small post (30 July 2026), main benchmark table, row “SciCode”, column “Inkling-Small”; the Hugging Face card (revision 8cc5877b44d3) repeats the table exactly. Thinking Machines states it uses Artificial Analysis’s reported score for this benchmark, for its own and other models; not an Artificial Analysis record captured by Examenos. Thinking Machines runs its own models at effort 0.99 (a 0–1 setting) and temperature 1.0; Effort 0.99 is stored as xhigh: Thinking Machines’ Inkling 2 Small post (5 November 2026, https://thinkingmachines.ai/news/inkling-2-small/) names this setting “effort 0.99 (xhigh)”; relabelled 7 November 2026 from unknown. Execution date unpublished.

1269 · raw 1269 Elo

Headline · xhigh effort · Own vendor

Source/record date: 2026-07-30

Thinking Machines’ Inkling-Small post (30 July 2026), main benchmark table, row “GDPval-AA v2”, column “Inkling-Small”; the Hugging Face card (revision 8cc5877b44d3) repeats the table exactly. Thinking Machines states it uses Artificial Analysis’s reported score for this benchmark, for its own and other models; not an Artificial Analysis record captured by Examenos. Thinking Machines runs its own models at effort 0.99 (a 0–1 setting) and temperature 1.0; Effort 0.99 is stored as xhigh: Thinking Machines’ Inkling 2 Small post (5 November 2026, https://thinkingmachines.ai/news/inkling-2-small/) names this setting “effort 0.99 (xhigh)”; relabelled 7 November 2026 from unknown. Execution date unpublished.

79.6% · raw 79.6 %

Headline · xhigh effort · Own vendor

Source/record date: 2026-07-30

Thinking Machines’ Inkling-Small post (30 July 2026), main benchmark table, row “MCP Atlaspublic / all”, column “Inkling-Small”; the Hugging Face card (revision 8cc5877b44d3) repeats the table exactly. Thinking Machines states it uses Scale AI’s reported score for this benchmark. Public task split. Thinking Machines runs its own models at effort 0.99 (a 0–1 setting) and temperature 1.0; Effort 0.99 is stored as xhigh: Thinking Machines’ Inkling 2 Small post (5 November 2026, https://thinkingmachines.ai/news/inkling-2-small/) names this setting “effort 0.99 (xhigh)”; relabelled 7 November 2026 from unknown. Execution date unpublished.

79.2% · raw 79.2 %

Headline · xhigh effort · Own vendor

Source/record date: 2026-07-30

Thinking Machines’ Inkling-Small post (30 July 2026), main benchmark table, row “MCP Atlaspublic / all”, column “Inkling-Small”; the Hugging Face card (revision 8cc5877b44d3) repeats the table exactly. Thinking Machines states it uses Scale AI’s reported score for this benchmark. All tasks. Thinking Machines runs its own models at effort 0.99 (a 0–1 setting) and temperature 1.0; Effort 0.99 is stored as xhigh: Thinking Machines’ Inkling 2 Small post (5 November 2026, https://thinkingmachines.ai/news/inkling-2-small/) names this setting “effort 0.99 (xhigh)”; relabelled 7 November 2026 from unknown. Execution date unpublished.

15.5% · raw 15.5 %

Headline · xhigh effort · Own vendor

Source/record date: 2026-07-30

Thinking Machines’ Inkling-Small post (30 July 2026), main benchmark table, row “Tau 3 Banking”, column “Inkling-Small”; the Hugging Face card (revision 8cc5877b44d3) repeats the table exactly. Thinking Machines states it uses Artificial Analysis’s reported score for this benchmark, for its own and other models; not an Artificial Analysis record captured by Examenos. Thinking Machines runs its own models at effort 0.99 (a 0–1 setting) and temperature 1.0; Effort 0.99 is stored as xhigh: Thinking Machines’ Inkling 2 Small post (5 November 2026, https://thinkingmachines.ai/news/inkling-2-small/) names this setting “effort 0.99 (xhigh)”; relabelled 7 November 2026 from unknown. Execution date unpublished.

77.4% · raw 77.4 %

Headline · xhigh effort · Own vendor

Source/record date: 2026-07-30

Thinking Machines’ Inkling-Small post (30 July 2026), main benchmark table, row “BrowseComp w/ ctx management”, column “Inkling-Small”; the Hugging Face card (revision 8cc5877b44d3) repeats the table exactly. With context management; harness unpublished. Thinking Machines runs its own models at effort 0.99 (a 0–1 setting) and temperature 1.0; Effort 0.99 is stored as xhigh: Thinking Machines’ Inkling 2 Small post (5 November 2026, https://thinkingmachines.ai/news/inkling-2-small/) names this setting “effort 0.99 (xhigh)”; relabelled 7 November 2026 from unknown. Execution date unpublished.

54.4% · raw 54.4 %

Headline · xhigh effort · Own vendor

Source/record date: 2026-07-30

Thinking Machines’ Inkling-Small post (30 July 2026), main benchmark table, row “Toolathlon Verified”, column “Inkling-Small”; the Hugging Face card (revision 8cc5877b44d3) repeats the table exactly. Harness unpublished. Thinking Machines runs its own models at effort 0.99 (a 0–1 setting) and temperature 1.0; Effort 0.99 is stored as xhigh: Thinking Machines’ Inkling 2 Small post (5 November 2026, https://thinkingmachines.ai/news/inkling-2-small/) names this setting “effort 0.99 (xhigh)”; relabelled 7 November 2026 from unknown. Execution date unpublished.

917 · raw 917 Elo

Headline · xhigh effort · Own vendor

Source/record date: 2026-07-30

Thinking Machines’ Inkling-Small post (30 July 2026), main benchmark table, row “AA-Briefcase”, column “Inkling-Small”; the Hugging Face card (revision 8cc5877b44d3) repeats the table exactly. Thinking Machines states it uses Artificial Analysis’s reported score for this benchmark, for its own and other models; not an Artificial Analysis record captured by Examenos. Elo-style rating. Thinking Machines runs its own models at effort 0.99 (a 0–1 setting) and temperature 1.0; Effort 0.99 is stored as xhigh: Thinking Machines’ Inkling 2 Small post (5 November 2026, https://thinkingmachines.ai/news/inkling-2-small/) names this setting “effort 0.99 (xhigh)”; relabelled 7 November 2026 from unknown. Execution date unpublished.

89.5% · raw 89.5 %

Headline · xhigh effort · Own vendor

Source/record date: 2026-07-30

Thinking Machines’ Inkling-Small post (30 July 2026), main benchmark table, row “GPQA Diamond”, column “Inkling-Small”; the Hugging Face card (revision 8cc5877b44d3) repeats the table exactly. Thinking Machines states it uses Artificial Analysis’s reported score for this benchmark, for its own and other models; not an Artificial Analysis record captured by Examenos. Thinking Machines runs its own models at effort 0.99 (a 0–1 setting) and temperature 1.0; Effort 0.99 is stored as xhigh: Thinking Machines’ Inkling 2 Small post (5 November 2026, https://thinkingmachines.ai/news/inkling-2-small/) names this setting “effort 0.99 (xhigh)”; relabelled 7 November 2026 from unknown. Execution date unpublished.

88.6% · raw 88.57 %

Alternative · unknown effort · Peer vendor

Source/record date: 2026-10-15

As reported by NVIDIA for the exact peer in its Nemotron 3.5 Super and Lightning technical report (15 October 2026), Table 10, column “Inkling Small”, row “GPQAD”; NVIDIA’s run, not a Thinking Machines result. NeMo Gym; report Table 10 and §§3.8–3.9. GPQA averages 8 repeats. Execution date and effort unpublished. Non-headline: the first-party row (89.5, https://thinkingmachines.ai/news/inkling-small/) holds this cell.

31.6% · raw 31.6 %

Headline · xhigh effort · Own vendor

Source/record date: 2026-07-30

Thinking Machines’ Inkling-Small post (30 July 2026), main benchmark table, row “HLE text only”, column “Inkling-Small”; the Hugging Face card (revision 8cc5877b44d3) repeats the table exactly. Thinking Machines states it uses Artificial Analysis’s reported score for this benchmark, for its own and other models; not an Artificial Analysis record captured by Examenos. Thinking Machines runs its own models at effort 0.99 (a 0–1 setting) and temperature 1.0; Effort 0.99 is stored as xhigh: Thinking Machines’ Inkling 2 Small post (5 November 2026, https://thinkingmachines.ai/news/inkling-2-small/) names this setting “effort 0.99 (xhigh)”; relabelled 7 November 2026 from unknown. Execution date unpublished.

33.3% · raw 33.27 %

Alternative · unknown effort · Peer vendor

Source/record date: 2026-10-15

As reported by NVIDIA for the exact peer in its Nemotron 3.5 Super and Lightning technical report (15 October 2026), Table 10, column “Inkling Small”, row “HLE(notools,text)”; NVIDIA’s run, not a Thinking Machines result. NeMo Gym; report Table 10 and §§3.8–3.9. HLE text-only has 2,158 questions. Execution date and effort unpublished. Non-headline: the first-party row (31.6, https://thinkingmachines.ai/news/inkling-small/) holds this cell.

47.8% · raw 47.8 %

Headline · xhigh effort · Own vendor

Source/record date: 2026-07-30

Thinking Machines’ Inkling-Small post (30 July 2026), main benchmark table, row “HLE with tools”, column “Inkling-Small”; the Hugging Face card (revision 8cc5877b44d3) repeats the table exactly. The post attributes Humanity’s Last Exam scores to Artificial Analysis but does not say whether that covers this with-tools row; tools and harness unpublished. Thinking Machines runs its own models at effort 0.99 (a 0–1 setting) and temperature 1.0; Effort 0.99 is stored as xhigh: Thinking Machines’ Inkling 2 Small post (5 November 2026, https://thinkingmachines.ai/news/inkling-2-small/) names this setting “effort 0.99 (xhigh)”; relabelled 7 November 2026 from unknown. Execution date unpublished.

95.5% · raw 95.5 %

Headline · xhigh effort · Own vendor

Source/record date: 2026-07-30

Thinking Machines’ Inkling-Small post (30 July 2026), main benchmark table, row “AIME 2026”, column “Inkling-Small”; the Hugging Face card (revision 8cc5877b44d3) repeats the table exactly. Sampling and repeats unpublished. Thinking Machines runs its own models at effort 0.99 (a 0–1 setting) and temperature 1.0; Effort 0.99 is stored as xhigh: Thinking Machines’ Inkling 2 Small post (5 November 2026, https://thinkingmachines.ai/news/inkling-2-small/) names this setting “effort 0.99 (xhigh)”; relabelled 7 November 2026 from unknown. Execution date unpublished.

90.2% · raw 90.2 %

Headline · xhigh effort · Own vendor

Source/record date: 2026-07-30

Thinking Machines’ Inkling-Small post (30 July 2026), main benchmark table, row “HMMT Feb 2026”, column “Inkling-Small”; the Hugging Face card (revision 8cc5877b44d3) repeats the table exactly. Sampling and repeats unpublished. Thinking Machines runs its own models at effort 0.99 (a 0–1 setting) and temperature 1.0; Effort 0.99 is stored as xhigh: Thinking Machines’ Inkling 2 Small post (5 November 2026, https://thinkingmachines.ai/news/inkling-2-small/) names this setting “effort 0.99 (xhigh)”; relabelled 7 November 2026 from unknown. Execution date unpublished.

8.3% · raw 8.3 %

Headline · xhigh effort · Own vendor

Source/record date: 2026-07-30

Thinking Machines’ Inkling-Small post (30 July 2026), main benchmark table, row “CritPt”, column “Inkling-Small”; the Hugging Face card (revision 8cc5877b44d3) repeats the table exactly. Harness unpublished. Thinking Machines runs its own models at effort 0.99 (a 0–1 setting) and temperature 1.0; Effort 0.99 is stored as xhigh: Thinking Machines’ Inkling 2 Small post (5 November 2026, https://thinkingmachines.ai/news/inkling-2-small/) names this setting “effort 0.99 (xhigh)”; relabelled 7 November 2026 from unknown. Execution date unpublished.

9.1% · raw 9.14 %

Alternative · unknown effort · Peer vendor

Source/record date: 2026-10-15

As reported by NVIDIA for the exact peer in its Nemotron 3.5 Super and Lightning technical report (15 October 2026), Table 10, column “Inkling Small”, row “CritPt”; NVIDIA’s run, not a Thinking Machines result. NeMo Gym; report Table 10 and §§3.8–3.9. Execution date and effort unpublished. Non-headline: the first-party row (8.3, https://thinkingmachines.ai/news/inkling-small/) holds this cell.

84% · raw 84 %

Headline · xhigh effort · Own vendor

Source/record date: 2026-07-30

Thinking Machines’ Inkling-Small post (30 July 2026), main benchmark table, row “ARC-AGI-1”, column “Inkling-Small”; the Hugging Face card (revision 8cc5877b44d3) repeats the table exactly. Thinking Machines states it uses ARC Prize’s reported score (verified semi-private set). Thinking Machines runs its own models at effort 0.99 (a 0–1 setting) and temperature 1.0; Effort 0.99 is stored as xhigh: Thinking Machines’ Inkling 2 Small post (5 November 2026, https://thinkingmachines.ai/news/inkling-2-small/) names this setting “effort 0.99 (xhigh)”; relabelled 7 November 2026 from unknown. Execution date unpublished.

40.1% · raw 40.1 %

Headline · xhigh effort · Own vendor

Source/record date: 2026-07-30

Thinking Machines’ Inkling-Small post (30 July 2026), main benchmark table, row “ARC-AGI-2”, column “Inkling-Small”; the Hugging Face card (revision 8cc5877b44d3) repeats the table exactly. Thinking Machines states it uses ARC Prize’s reported score (verified semi-private set). Thinking Machines runs its own models at effort 0.99 (a 0–1 setting) and temperature 1.0; Effort 0.99 is stored as xhigh: Thinking Machines’ Inkling 2 Small post (5 November 2026, https://thinkingmachines.ai/news/inkling-2-small/) names this setting “effort 0.99 (xhigh)”; relabelled 7 November 2026 from unknown. Execution date unpublished.

20.6% · raw 20.6 %

Headline · xhigh effort · Own vendor

Source/record date: 2026-07-30

Thinking Machines’ Inkling-Small post (30 July 2026), main benchmark table, row “SimpleQA Verified”, column “Inkling-Small”; the Hugging Face card (revision 8cc5877b44d3) repeats the table exactly. Grader unpublished. Thinking Machines runs its own models at effort 0.99 (a 0–1 setting) and temperature 1.0; Effort 0.99 is stored as xhigh: Thinking Machines’ Inkling 2 Small post (5 November 2026, https://thinkingmachines.ai/news/inkling-2-small/) names this setting “effort 0.99 (xhigh)”; relabelled 7 November 2026 from unknown. Execution date unpublished.

-9 · raw -9 index

Headline · xhigh effort · Own vendor

Source/record date: 2026-07-30

Thinking Machines’ Inkling-Small post (30 July 2026), main benchmark table, row “AA Omniscience”, column “Inkling-Small”; the Hugging Face card (revision 8cc5877b44d3) repeats the table exactly. Thinking Machines states it uses Artificial Analysis’s reported score for this benchmark, for its own and other models; not an Artificial Analysis record captured by Examenos. Omniscience index (−100 to 100), not an accuracy. Thinking Machines runs its own models at effort 0.99 (a 0–1 setting) and temperature 1.0; Effort 0.99 is stored as xhigh: Thinking Machines’ Inkling 2 Small post (5 November 2026, https://thinkingmachines.ai/news/inkling-2-small/) names this setting “effort 0.99 (xhigh)”; relabelled 7 November 2026 from unknown. Execution date unpublished.

82.2% · raw 82.2 %

Headline · xhigh effort · Own vendor

Source/record date: 2026-07-30

Thinking Machines’ Inkling-Small post (30 July 2026), main benchmark table, row “IFBench”, column “Inkling-Small”; the Hugging Face card (revision 8cc5877b44d3) repeats the table exactly. Prompt or instruction level and strict or loose scoring unpublished. Thinking Machines runs its own models at effort 0.99 (a 0–1 setting) and temperature 1.0; Effort 0.99 is stored as xhigh: Thinking Machines’ Inkling 2 Small post (5 November 2026, https://thinkingmachines.ai/news/inkling-2-small/) names this setting “effort 0.99 (xhigh)”; relabelled 7 November 2026 from unknown. Execution date unpublished.

86.7% · raw 86.7 %

Headline · xhigh effort · Own vendor

Source/record date: 2026-07-30

Thinking Machines’ Inkling-Small post (30 July 2026), main benchmark table, row “Global-MMLU-Lite”, column “Inkling-Small”; the Hugging Face card (revision 8cc5877b44d3) repeats the table exactly. Prompting unpublished. Thinking Machines runs its own models at effort 0.99 (a 0–1 setting) and temperature 1.0; Effort 0.99 is stored as xhigh: Thinking Machines’ Inkling 2 Small post (5 November 2026, https://thinkingmachines.ai/news/inkling-2-small/) names this setting “effort 0.99 (xhigh)”; relabelled 7 November 2026 from unknown. Execution date unpublished.

98.4% · raw 98.4 %

Headline · xhigh effort · Own vendor

Source/record date: 2026-07-30

Thinking Machines’ Inkling-Small post (30 July 2026), main benchmark table, row “StrongREJECT”, column “Inkling-Small”; the Hugging Face card (revision 8cc5877b44d3) repeats the table exactly. Higher is better. Thinking Machines runs its own models at effort 0.99 (a 0–1 setting) and temperature 1.0; Effort 0.99 is stored as xhigh: Thinking Machines’ Inkling 2 Small post (5 November 2026, https://thinkingmachines.ai/news/inkling-2-small/) names this setting “effort 0.99 (xhigh)”; relabelled 7 November 2026 from unknown. Execution date unpublished.

71.6% · raw 71.6 %

Headline · xhigh effort · Own vendor

Source/record date: 2026-07-30

Thinking Machines’ Inkling-Small post (30 July 2026), main benchmark table, row “FORTRESS Adversarial”, column “Inkling-Small”; the Hugging Face card (revision 8cc5877b44d3) repeats the table exactly. Higher is better: the rate of refusing harmful requests. Thinking Machines runs its own models at effort 0.99 (a 0–1 setting) and temperature 1.0; Effort 0.99 is stored as xhigh: Thinking Machines’ Inkling 2 Small post (5 November 2026, https://thinkingmachines.ai/news/inkling-2-small/) names this setting “effort 0.99 (xhigh)”; relabelled 7 November 2026 from unknown. Execution date unpublished.

96.9% · raw 96.9 %

Headline · xhigh effort · Own vendor

Source/record date: 2026-07-30

Thinking Machines’ Inkling-Small post (30 July 2026), main benchmark table, row “FORTRESS Benign”, column “Inkling-Small”; the Hugging Face card (revision 8cc5877b44d3) repeats the table exactly. Higher is better: the rate of still answering safe look-alike requests. Thinking Machines runs its own models at effort 0.99 (a 0–1 setting) and temperature 1.0; Effort 0.99 is stored as xhigh: Thinking Machines’ Inkling 2 Small post (5 November 2026, https://thinkingmachines.ai/news/inkling-2-small/) names this setting “effort 0.99 (xhigh)”; relabelled 7 November 2026 from unknown. Execution date unpublished.

74% · raw 74 %

Headline · xhigh effort · Own vendor

Source/record date: 2026-07-30

Thinking Machines’ Inkling-Small post (30 July 2026), main benchmark table, row “MMMU Pro Standard 10”, column “Inkling-Small”; the Hugging Face card (revision 8cc5877b44d3) repeats the table exactly. Thinking Machines states it uses Artificial Analysis’s reported score for this benchmark, for its own and other models; not an Artificial Analysis record captured by Examenos. Standard setting with 10 options. Thinking Machines runs its own models at effort 0.99 (a 0–1 setting) and temperature 1.0; Effort 0.99 is stored as xhigh: Thinking Machines’ Inkling 2 Small post (5 November 2026, https://thinkingmachines.ai/news/inkling-2-small/) names this setting “effort 0.99 (xhigh)”; relabelled 7 November 2026 from unknown. Execution date unpublished.

77.4% · raw 77.4 %

Headline · xhigh effort · Own vendor

Source/record date: 2026-07-30

Thinking Machines’ Inkling-Small post (30 July 2026), main benchmark table, row “Charxiv RQoriginal / with python”, column “Inkling-Small”; the Hugging Face card (revision 8cc5877b44d3) repeats the table exactly. No tools. Thinking Machines runs its own models at effort 0.99 (a 0–1 setting) and temperature 1.0; Effort 0.99 is stored as xhigh: Thinking Machines’ Inkling 2 Small post (5 November 2026, https://thinkingmachines.ai/news/inkling-2-small/) names this setting “effort 0.99 (xhigh)”; relabelled 7 November 2026 from unknown. Execution date unpublished.

77.8% · raw 77.77 %

Alternative · unknown effort · Peer vendor

Source/record date: 2026-10-15

As reported by NVIDIA for the exact peer in its Nemotron 3.5 Super and Lightning technical report (15 October 2026), Table 10, column “Inkling Small”, row “CharXiv-RQ”; NVIDIA’s run, not a Thinking Machines result. NeMo Gym; report Table 10 and §§3.8–3.9. Execution date and effort unpublished. Non-headline: the first-party row (77.4, https://thinkingmachines.ai/news/inkling-small/) holds this cell.

81.3% · raw 81.3 %

Headline · xhigh effort · Own vendor

Source/record date: 2026-07-30

Thinking Machines’ Inkling-Small post (30 July 2026), main benchmark table, row “Charxiv RQoriginal / with python”, column “Inkling-Small”; the Hugging Face card (revision 8cc5877b44d3) repeats the table exactly. With a Python tool. Thinking Machines runs its own models at effort 0.99 (a 0–1 setting) and temperature 1.0; Effort 0.99 is stored as xhigh: Thinking Machines’ Inkling 2 Small post (5 November 2026, https://thinkingmachines.ai/news/inkling-2-small/) names this setting “effort 0.99 (xhigh)”; relabelled 7 November 2026 from unknown. Execution date unpublished.

54.9% · raw 54.9 %

Headline · xhigh effort · Own vendor

Source/record date: 2026-07-30

Thinking Machines’ Inkling-Small post (30 July 2026), main benchmark table, row “Audio MC”, column “Inkling-Small”; the Hugging Face card (revision 8cc5877b44d3) repeats the table exactly. Thinking Machines states it uses Scale AI’s reported score for Audio MultiChallenge (named in the Inkling-Small post). Thinking Machines runs its own models at effort 0.99 (a 0–1 setting) and temperature 1.0; Effort 0.99 is stored as xhigh: Thinking Machines’ Inkling 2 Small post (5 November 2026, https://thinkingmachines.ai/news/inkling-2-small/) names this setting “effort 0.99 (xhigh)”; relabelled 7 November 2026 from unknown. Execution date unpublished.

77% · raw 77 %

Headline · xhigh effort · Own vendor

Source/record date: 2026-07-30

Thinking Machines’ Inkling-Small post (30 July 2026), main benchmark table, row “MMAU”, column “Inkling-Small”; the Hugging Face card (revision 8cc5877b44d3) repeats the table exactly. Split unpublished. Thinking Machines runs its own models at effort 0.99 (a 0–1 setting) and temperature 1.0; Effort 0.99 is stored as xhigh: Thinking Machines’ Inkling 2 Small post (5 November 2026, https://thinkingmachines.ai/news/inkling-2-small/) names this setting “effort 0.99 (xhigh)”; relabelled 7 November 2026 from unknown. Execution date unpublished.

90.1% · raw 90.1 %

Headline · xhigh effort · Own vendor

Source/record date: 2026-07-30

Thinking Machines’ Inkling-Small post (30 July 2026), main benchmark table, row “VoiceBench”, column “Inkling-Small”; the Hugging Face card (revision 8cc5877b44d3) repeats the table exactly. Thinking Machines added a system message telling models to follow the expected answer format, because VoiceBench grades by string matching. Thinking Machines runs its own models at effort 0.99 (a 0–1 setting) and temperature 1.0; Effort 0.99 is stored as xhigh: Thinking Machines’ Inkling 2 Small post (5 November 2026, https://thinkingmachines.ai/news/inkling-2-small/) names this setting “effort 0.99 (xhigh)”; relabelled 7 November 2026 from unknown. Execution date unpublished.

61.3 · raw 61.3 index

Headline · xhigh effort · Own vendor

Source/record date: 2026-07-30

Thinking Machines’ Inkling-Small post (30 July 2026), forecasting table, column “Inkling-Small”: Forecasting Research Institute ForecastBench, no search, Brier Index (higher is better), 61.3 ± 0.46. Thinking Machines names the Forecasting Research Institute and ProphetArena as the sources of these scores; results were obtained during testing between 19 and 28 July 2026. The ± value (0.46) is the post’s published uncertainty, interval type unstated. Thinking Machines runs its own models at effort 0.99 (a 0–1 setting) and temperature 1.0; Effort 0.99 is stored as xhigh: Thinking Machines’ Inkling 2 Small post (5 November 2026, https://thinkingmachines.ai/news/inkling-2-small/) names this setting “effort 0.99 (xhigh)”; relabelled 7 November 2026 from unknown.

61.5 · raw 61.5 index

Headline · xhigh effort · Own vendor

Source/record date: 2026-07-30

Thinking Machines’ Inkling-Small post (30 July 2026), forecasting table, column “Inkling-Small”: Forecasting Research Institute ForecastBench, with search, Brier Index (higher is better), 61.5 ± 0.54. Thinking Machines names the Forecasting Research Institute and ProphetArena as the sources of these scores; results were obtained during testing between 19 and 28 July 2026. The ± value (0.54) is the post’s published uncertainty, interval type unstated. Thinking Machines runs its own models at effort 0.99 (a 0–1 setting) and temperature 1.0; Effort 0.99 is stored as xhigh: Thinking Machines’ Inkling 2 Small post (5 November 2026, https://thinkingmachines.ai/news/inkling-2-small/) names this setting “effort 0.99 (xhigh)”; relabelled 7 November 2026 from unknown.

0.1 score · raw 0.1238 score

Headline · xhigh effort · Own vendor

Source/record date: 2026-07-30

Thinking Machines’ Inkling-Small post (30 July 2026), forecasting table, column “Inkling-Small”: Prophet Arena Brier score (lower is better), 0.1238 ± 0.0086. Thinking Machines names the Forecasting Research Institute and ProphetArena as the sources of these scores; results were obtained during testing between 19 and 28 July 2026. The ± value (0.0086) is the post’s published uncertainty, interval type unstated. Thinking Machines runs its own models at effort 0.99 (a 0–1 setting) and temperature 1.0; Effort 0.99 is stored as xhigh: Thinking Machines’ Inkling 2 Small post (5 November 2026, https://thinkingmachines.ai/news/inkling-2-small/) names this setting “effort 0.99 (xhigh)”; relabelled 7 November 2026 from unknown.

75.3% · raw 75.27 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-15

As reported by NVIDIA for the exact peer in its Nemotron 3.5 Super and Lightning technical report (15 October 2026), Table 10, column “Inkling Small”, row “SWE-benchVerified”; NVIDIA’s run, not a Thinking Machines result. NeMo Gym; report Table 10 and §§3.8–3.9. All SWE evaluations use OpenCode as the default harness. Execution date and effort unpublished.

71.4% · raw 71.44 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-15

As reported by NVIDIA for the exact peer in its Nemotron 3.5 Super and Lightning technical report (15 October 2026), Table 10, column “Inkling Small”, row “SWE-benchMultilingual”; NVIDIA’s run, not a Thinking Machines result. NeMo Gym; report Table 10 and §§3.8–3.9. All SWE evaluations use OpenCode as the default harness. Execution date and effort unpublished.

56.5% · raw 56.54 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-15

As reported by NVIDIA for the exact peer in its Nemotron 3.5 Super and Lightning technical report (15 October 2026), Table 10, column “Inkling Small”, row “SWE-benchPro”; NVIDIA’s run, not a Thinking Machines result. NeMo Gym; report Table 10 and §§3.8–3.9. All SWE evaluations use OpenCode as the default harness. Execution date and effort unpublished.

71.8% · raw 71.82 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-15

As reported by NVIDIA for the exact peer in its Nemotron 3.5 Super and Lightning technical report (15 October 2026), Table 10, column “Inkling Small”, row “Tau2Retail”; NVIDIA’s run, not a Thinking Machines result. NeMo Gym; report Table 10 and §§3.8–3.9. Execution date and effort unpublished.

62.8% · raw 62.75 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-15

As reported by NVIDIA for the exact peer in its Nemotron 3.5 Super and Lightning technical report (15 October 2026), Table 10, column “Inkling Small”, row “Tau2Airline”; NVIDIA’s run, not a Thinking Machines result. NeMo Gym; report Table 10 and §§3.8–3.9. Execution date and effort unpublished.

90.1% · raw 90.12 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-15

As reported by NVIDIA for the exact peer in its Nemotron 3.5 Super and Lightning technical report (15 October 2026), Table 10, column “Inkling Small”, row “Tau2Telecom”; NVIDIA’s run, not a Thinking Machines result. NeMo Gym; report Table 10 and §§3.8–3.9. Execution date and effort unpublished.

1313.3 · raw 1313.27 Elo

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-15

As reported by NVIDIA for the exact peer in its Nemotron 3.5 Super and Lightning technical report (15 October 2026), Table 10, column “Inkling Small”, row “GDPVal-AA-v2ELO”; NVIDIA’s run, not a Thinking Machines result. NeMo Gym; report Table 10 and §§3.8–3.9. Stirrup agent, NVIDIA reference committee loosely following AA v2.1; not independent AA Elo. Execution date and effort unpublished.

40.7% · raw 40.7 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-15

As reported by NVIDIA for the exact peer in its Nemotron 3.5 Super and Lightning technical report (15 October 2026), Table 10, column “Inkling Small”, row “GDPVal-AA-v2normELO”; NVIDIA’s run, not a Thinking Machines result. NeMo Gym; report Table 10 and §§3.8–3.9. Stirrup agent, NVIDIA reference committee loosely following AA v2.1; not independent AA Elo. Raw source normalized value ×100; not a pass rate. Execution date and effort unpublished.

74.1% · raw 74.1 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-15

As reported by NVIDIA for the exact peer in its Nemotron 3.5 Super and Lightning technical report (15 October 2026), Table 10, column “Inkling Small”, row “BrowseComp”; NVIDIA’s run, not a Thinking Machines result. NeMo Gym; report Table 10 and §§3.8–3.9. NeMo Gym BrowseComp with Exa search. Execution date and effort unpublished.

55.4% · raw 55.37 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-15

As reported by NVIDIA for the exact peer in its Nemotron 3.5 Super and Lightning technical report (15 October 2026), Table 10, column “Inkling Small”, row “BrowseComp(OpenCode)”; NVIDIA’s run, not a Thinking Machines result. NeMo Gym; report Table 10 and §§3.8–3.9. OpenCode harness. Execution date and effort unpublished.

48.3% · raw 48.28 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-15

As reported by NVIDIA for the exact peer in its Nemotron 3.5 Super and Lightning technical report (15 October 2026), Table 10, column “Inkling Small”, row “SciCode(subtask)”; NVIDIA’s run, not a Thinking Machines result. NeMo Gym; report Table 10 and §§3.8–3.9. SciCode uses scientist background / 8 repeats; subtask accuracy. Execution date and effort unpublished.

18.7% · raw 18.65 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-15

As reported by NVIDIA for the exact peer in its Nemotron 3.5 Super and Lightning technical report (15 October 2026), Table 10, column “Inkling Small”, row “SciCode(problem)”; NVIDIA’s run, not a Thinking Machines result. NeMo Gym; report Table 10 and §§3.8–3.9. SciCode uses scientist background / 8 repeats; whole-problem accuracy. Execution date and effort unpublished.

88.3% · raw 88.25 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-15

As reported by NVIDIA for the exact peer in its Nemotron 3.5 Super and Lightning technical report (15 October 2026), Table 10, column “Inkling Small”, row “LiveCodeBenchv6”; NVIDIA’s run, not a Thinking Machines result. NeMo Gym; report Table 10 and §§3.8–3.9. Execution date and effort unpublished.

71.5% · raw 71.54 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-15

As reported by NVIDIA for the exact peer in its Nemotron 3.5 Super and Lightning technical report (15 October 2026), Table 10, column “Inkling Small”, row “Apex-Shortlist”; NVIDIA’s run, not a Thinking Machines result. NeMo Gym; report Table 10 and §§3.8–3.9. APEX Shortlist 16 repeats, no tools. Execution date and effort unpublished.

35.4% · raw 35.38 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-15

As reported by NVIDIA for the exact peer in its Nemotron 3.5 Super and Lightning technical report (15 October 2026), Table 10, column “Inkling Small”, row “Accuracy”; NVIDIA’s run, not a Thinking Machines result. NeMo Gym; report Table 10 and §§3.8–3.9. AA Omniscience uses 600 public questions / 10 repeats with GPT-5.6 Luna judge. Execution date and effort unpublished.

40.4% · raw 40.39 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-15

As reported by NVIDIA for the exact peer in its Nemotron 3.5 Super and Lightning technical report (15 October 2026), Table 10, column “Inkling Small”, row “Non-Hallucination”; NVIDIA’s run, not a Thinking Machines result. NeMo Gym; report Table 10 and §§3.8–3.9. AA Omniscience uses 600 public questions / 10 repeats with GPT-5.6 Luna judge. Execution date and effort unpublished.

24.7% · raw 24.74 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-15

As reported by NVIDIA for the exact peer in its Nemotron 3.5 Super and Lightning technical report (15 October 2026), Table 10, column “Inkling Small”, row “BabyVision”; NVIDIA’s run, not a Thinking Machines result. NeMo Gym; report Table 10 and §§3.8–3.9. Execution date and effort unpublished.

78.8% · raw 78.77 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-15

As reported by NVIDIA for the exact peer in its Nemotron 3.5 Super and Lightning technical report (15 October 2026), Table 10, column “Inkling Small”, row “MathVision”; NVIDIA’s run, not a Thinking Machines result. NeMo Gym; report Table 10 and §§3.8–3.9. Execution date and effort unpublished.

54.7% · raw 54.69 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-15

As reported by NVIDIA for the exact peer in its Nemotron 3.5 Super and Lightning technical report (15 October 2026), Table 10, column “Inkling Small”, row “OCR-Reasoning”; NVIDIA’s run, not a Thinking Machines result. NeMo Gym; report Table 10 and §§3.8–3.9. Execution date and effort unpublished.

54.2% · raw 54.21 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-15

As reported by NVIDIA for the exact peer in its Nemotron 3.5 Super and Lightning technical report (15 October 2026), Table 10, column “Inkling Small”, row “OCRBench-v2(EN)”; NVIDIA’s run, not a Thinking Machines result. NeMo Gym; report Table 10 and §§3.8–3.9. Execution date and effort unpublished.

52.4% · raw 52.38 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-15

As reported by NVIDIA for the exact peer in its Nemotron 3.5 Super and Lightning technical report (15 October 2026), Table 10, column “Inkling Small”, row “OCRBench-v2(ZH)”; NVIDIA’s run, not a Thinking Machines result. NeMo Gym; report Table 10 and §§3.8–3.9. Execution date and effort unpublished.

70.5% · raw 70.51 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-15

As reported by NVIDIA for the exact peer in its Nemotron 3.5 Super and Lightning technical report (15 October 2026), Table 10, column “Inkling Small”, row “V*”; NVIDIA’s run, not a Thinking Machines result. NeMo Gym; report Table 10 and §§3.8–3.9. V* has no search tool. Execution date and effort unpublished.

49.1% · raw 49.06 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-15

As reported by NVIDIA for the exact peer in its Nemotron 3.5 Super and Lightning technical report (15 October 2026), Table 10, column “Inkling Small”, row “MMLongBench-Doc”; NVIDIA’s run, not a Thinking Machines result. NeMo Gym; report Table 10 and §§3.8–3.9. Execution date and effort unpublished.

77.3% · raw 77.31 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-15

As reported by NVIDIA for the exact peer in its Nemotron 3.5 Super and Lightning technical report (15 October 2026), Table 10, column “Inkling Small”, row “AA-LCR”; NVIDIA’s run, not a Thinking Machines result. NeMo Gym; report Table 10 and §§3.8–3.9. Execution date and effort unpublished.

55.7% · raw 55.74 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-15

As reported by NVIDIA for the exact peer in its Nemotron 3.5 Super and Lightning technical report (15 October 2026), Table 10, column “Inkling Small”, row “GraphWalks(≤1M)”; NVIDIA’s run, not a Thinking Machines result. NeMo Gym; report Table 10 and §§3.8–3.9. GraphWalks averages F1 across BFS and parents on the same 950/1,150 prompts fitting ≤1M. Execution date and effort unpublished.

79.3% · raw 79.31 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-15

As reported by NVIDIA for the exact peer in its Nemotron 3.5 Super and Lightning technical report (15 October 2026), Table 10, column “Inkling Small”, row “IFBench”; NVIDIA’s run, not a Thinking Machines result. NeMo Gym; report Table 10 and §§3.8–3.9. IFBench is mean fraction of constraints satisfied under strict verification, not prompt all-pass. Execution date and effort unpublished.

53.6% · raw 53.6 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-15

As reported by NVIDIA for the exact peer in its Nemotron 3.5 Super and Lightning technical report (15 October 2026), Table 10, column “Inkling Small”, row “NVChatBenchmark”; NVIDIA’s run, not a Thinking Machines result. NeMo Gym; report Table 10 and §§3.8–3.9. NVIDIA style-controlled chat preference win rate. Execution date and effort unpublished.

84% · raw 83.97 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-15

As reported by NVIDIA for the exact peer in its Nemotron 3.5 Super and Lightning technical report (15 October 2026), Table 10, column “Inkling Small”, row “MMLU-ProX”; NVIDIA’s run, not a Thinking Machines result. NeMo Gym; report Table 10 and §§3.8–3.9. Execution date and effort unpublished.

84.6 score · raw 84.6 score

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-15

As reported by NVIDIA for the exact peer in its Nemotron 3.5 Super and Lightning technical report (15 October 2026), Table 10, column “Inkling Small”, row “WMT24ppCOMET”; NVIDIA’s run, not a Thinking Machines result. NeMo Gym; report Table 10 and §§3.8–3.9. XCOMET-XXL, English to nine languages. Execution date and effort unpublished.

59.6% · raw 59.6 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-15

As reported by NVIDIA for the exact peer in the Nemotron 3.5 Super VL BF16 card (15 October 2026), column “Inkling-Small”; NVIDIA’s run, not a Thinking Machines result. Exact released BF16 card table. Conflicts/rounding versus detailed report remain non-headline alternatives; detailed report supplies the precise protocol. Raw card label: AA-Omniscience hallucination rate. Execution date and effort unpublished.

-3.1 · raw -3.1 index

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-15

As reported by NVIDIA for the exact peer in the Nemotron 3.5 Super VL BF16 card (15 October 2026), column “Inkling-Small”; NVIDIA’s run, not a Thinking Machines result. Exact released BF16 card table. Conflicts/rounding versus detailed report remain non-headline alternatives; detailed report supplies the precise protocol. Raw card label: AA-Omniscience index. Execution date and effort unpublished.

74% · raw 74 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-15

As reported by NVIDIA for the exact peer in the Nemotron 3.5 Super VL BF16 card (15 October 2026), column “Inkling-Small”; NVIDIA’s run, not a Thinking Machines result. Exact card offline snapshot; full source text: 74. Tuple labels and parentheses are not explained; no quantity/denominator inferred. Reported external AA values are quotes, not independent verification. Execution date and effort unpublished.

5.3% · raw 5.25 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-15

As reported by NVIDIA for the exact peer in the Nemotron 3.5 Super VL BF16 card (15 October 2026), column “Inkling-Small”; NVIDIA’s run, not a Thinking Machines result. Exact card offline snapshot; full source text: 5.25. Tuple labels and parentheses are not explained; no quantity/denominator inferred. Reported external AA values are quotes, not independent verification. Execution date and effort unpublished.

765 score · raw 765 score

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-15

As reported by NVIDIA for the exact peer in the Nemotron 3.5 Super VL BF16 card (15 October 2026), column “Inkling-Small”; NVIDIA’s run, not a Thinking Machines result. Exact card offline snapshot; full source text: 765 | 684 | 17. Tuple labels and parentheses are not explained; no quantity/denominator inferred. Reported external AA values are quotes, not independent verification. Execution date and effort unpublished.

684 score · raw 684 score

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-15

As reported by NVIDIA for the exact peer in the Nemotron 3.5 Super VL BF16 card (15 October 2026), column “Inkling-Small”; NVIDIA’s run, not a Thinking Machines result. Exact card offline snapshot; full source text: 765 | 684 | 17. Tuple labels and parentheses are not explained; no quantity/denominator inferred. Reported external AA values are quotes, not independent verification. Execution date and effort unpublished.

17 score · raw 17 score

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-15

As reported by NVIDIA for the exact peer in the Nemotron 3.5 Super VL BF16 card (15 October 2026), column “Inkling-Small”; NVIDIA’s run, not a Thinking Machines result. Exact card offline snapshot; full source text: 765 | 684 | 17. Tuple labels and parentheses are not explained; no quantity/denominator inferred. Reported external AA values are quotes, not independent verification. Execution date and effort unpublished.

32.8% · raw 32.83 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-15

As reported by NVIDIA for the exact peer in the Nemotron 3.5 Super VL BF16 card (15 October 2026), column “Inkling-Small”; NVIDIA’s run, not a Thinking Machines result. Exact card offline snapshot; full source text: 32.83. Tuple labels and parentheses are not explained; no quantity/denominator inferred. Reported external AA values are quotes, not independent verification. Execution date and effort unpublished.

8.6% · raw 8.6 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-15

As reported by NVIDIA for the exact peer in the Nemotron 3.5 Super VL BF16 card (15 October 2026), column “Inkling-Small”; NVIDIA’s run, not a Thinking Machines result. Exact card offline snapshot; full source text: 8.6 (AA 11.2). Tuple labels and parentheses are not explained; no quantity/denominator inferred. Reported external AA values are quotes, not independent verification. Execution date and effort unpublished.

0.5% · raw 0.5 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-15

As reported by NVIDIA for the exact peer in the Nemotron 3.5 Super VL BF16 card (15 October 2026), column “Inkling-Small”; NVIDIA’s run, not a Thinking Machines result. Exact card offline snapshot; full source text: 0.5. Tuple labels and parentheses are not explained; no quantity/denominator inferred. Reported external AA values are quotes, not independent verification. Execution date and effort unpublished.

18% · raw 18 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-15

As reported by NVIDIA for the exact peer in the Nemotron 3.5 Super VL BF16 card (15 October 2026), column “Inkling-Small”; NVIDIA’s run, not a Thinking Machines result. Exact card offline snapshot; full source text: 18. Tuple labels and parentheses are not explained; no quantity/denominator inferred. Reported external AA values are quotes, not independent verification. Execution date and effort unpublished.

1.3% · raw 1.27 %

Alternative · high effort · Own vendor

Source/record date: 2026-07-30

Cognition primary public data.json (https://cognition.com/data/frontier-code-2.0/data.json), FrontierCode 2.0 main, 132 tasks; weighted new_score x100, not unweighted correct. Harness: devin-cli. {"correct": 0.0154, "tokens": 13658, "cost": 1.366, "flagged_rate": 0.0256}. Cost is source-observed mean USD per rollout at list API pricing, not an API rate. Data checked 5 November; evaluation date unpublished. Headline is the best-scoring published effort, as Cognition displays it. Board effort labels for Inkling-Small map Thinking Machines’ 0–1 setting to Cognition’s names; Cognition does not publish the numeric setting.

3.2% · raw 3.21 %

Headline · max effort · Own vendor

Source/record date: 2026-07-30

Cognition primary public data.json (https://cognition.com/data/frontier-code-2.0/data.json), FrontierCode 2.0 main, 132 tasks; weighted new_score x100, not unweighted correct. Harness: devin-cli. {"correct": 0.0379, "tokens": 39077, "cost": 3.684, "flagged_rate": 0.0606}. Cost is source-observed mean USD per rollout at list API pricing, not an API rate. Data checked 5 November; evaluation date unpublished. Headline is the best-scoring published effort, as Cognition displays it. Board effort labels for Inkling-Small map Thinking Machines’ 0–1 setting to Cognition’s names; Cognition does not publish the numeric setting.

21.5% · raw 21.5 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-11-04

As reported by Cognition in its SWE-2 Flash launch table (4 November 2026), row “FrontierCode 1.1 Main”, column “Inkling-Small”; not a result from the model’s developer. Cognition publishes no methodology for this table; its SWE-2 post says it reports a model’s public result where one exists and otherwise runs it in the harness it was developed for (Claude Code for Anthropic, Codex for OpenAI, Devin CLI for open-weight models) at its best reasoning effort. Effort unstated. SWE-2 Flash is trained on top of Inkling-Small, so this is Cognition’s base-model comparison.

17.4% · raw 17.4 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-11-04

As reported by Cognition in its SWE-2 Flash launch table (4 November 2026), row “DeepSWE 1.1”, column “Inkling-Small”; not a result from the model’s developer. Cognition publishes no methodology for this table; its SWE-2 post says it reports a model’s public result where one exists and otherwise runs it in the harness it was developed for (Claude Code for Anthropic, Codex for OpenAI, Devin CLI for open-weight models) at its best reasoning effort. Effort unstated. SWE-2 Flash is trained on top of Inkling-Small, so this is Cognition’s base-model comparison.

0.9% · raw 0.9 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-11-04

As reported by Cognition in its SWE-2 Flash launch table (4 November 2026), row “Terminal-Bench 4.0”, column “Inkling-Small”; not a result from the model’s developer. Cognition publishes no methodology for this table; its SWE-2 post says it reports a model’s public result where one exists and otherwise runs it in the harness it was developed for (Claude Code for Anthropic, Codex for OpenAI, Devin CLI for open-weight models) at its best reasoning effort. Effort unstated. SWE-2 Flash is trained on top of Inkling-Small, so this is Cognition’s base-model comparison.

27.4% · raw 27.4 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-11-04

As reported by Cognition in its SWE-2 Flash launch table (4 November 2026), row “SWE-Atlas QA”, column “Inkling-Small”; not a result from the model’s developer. Cognition labels it “SWE-Atlas QA”. Cognition publishes no methodology for this table; its SWE-2 post says it reports a model’s public result where one exists and otherwise runs it in the harness it was developed for (Claude Code for Anthropic, Codex for OpenAI, Devin CLI for open-weight models) at its best reasoning effort. Effort unstated. SWE-2 Flash is trained on top of Inkling-Small, so this is Cognition’s base-model comparison.

56.5% · raw 56.5 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-11-04

As reported by Cognition in its SWE-2 Flash launch table (4 November 2026), row “SWE-Bench Pro v2 Hard”, column “Inkling-Small”; not a result from the model’s developer. Cognition publishes no methodology for this table; its SWE-2 post says it reports a model’s public result where one exists and otherwise runs it in the harness it was developed for (Claude Code for Anthropic, Codex for OpenAI, Devin CLI for open-weight models) at its best reasoning effort. Effort unstated. SWE-2 Flash is trained on top of Inkling-Small, so this is Cognition’s base-model comparison.

11.3% · raw 11.3 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-11-04

As reported by Cognition in its SWE-2 Flash launch table (4 November 2026), row “ExploitBench”, column “Inkling-Small”; not a result from the model’s developer. Cognition publishes no methodology for this table; its SWE-2 post says it reports a model’s public result where one exists and otherwise runs it in the harness it was developed for (Claude Code for Anthropic, Codex for OpenAI, Devin CLI for open-weight models) at its best reasoning effort. Effort unstated. SWE-2 Flash is trained on top of Inkling-Small, so this is Cognition’s base-model comparison.

27.5% · raw 27.5 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-02

pass^5, the share of 80 tasks solved on all 5 attempts; not pass@1. AA's label states no reasoning effort. Manual observation of Artificial Analysis's public AA-AnalystAgent evaluation page, "AA-AnalystAgent: Score" chart, 2 November 2026 (all 51 listed configurations selected; rendered bar label paired with the x-axis model link; research-raw/aa-analyst-mlcr-2026-11-02/cap_analystagent_score.txt). Configuration "Inkling Small". AA's displayed rounding; no API, download, hidden precision or inferred run date. Added 2026-11-05 when the model was added to Examenos.

7.8% · raw 7.8 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-02

Pass rate on AA's private held-out expert and compound set, 60 questions run 3 times each. AA's label states no reasoning effort. Manual observation of Artificial Analysis's public MLCR-AA evaluation page, "MLCR-AA: Score" chart, 2 November 2026 (all 176 listed configurations selected; rendered bar label paired with the x-axis model link; research-raw/aa-analyst-mlcr-2026-11-02/cap_mlcr_score.txt). Configuration "Inkling Small". AA's displayed rounding; no API, download, hidden precision or inferred run date. Added 2026-11-05 when the model was added to Examenos.

13 · raw 13 index

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Intelligence Index v5.0.1 (AA changelog, 3 November 2026: v5.0 with CritPt v2 regraded); never compared with v5.0 or v4.3.2 rows. AA's label states no reasoning effort. Manual observation of Artificial Analysis's public Intelligence Index evaluation page (title "Artificial Analysis Intelligence Index v5.0.1"), "Artificial Analysis Intelligence Index: Score" chart, 5 November 2026 (106 configurations selected with ?models=; rendered bar label paired with the x-axis model link; research-raw/aa-v5.0.1-2026-11-05/cap_index_score.txt). Configuration "Inkling Small". AA's displayed rounding; no API, download, hidden precision or inferred run date.

$0.2/task · raw 0.2 USD/task

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

AA usage-priced cost per task at its list pricing; not an API tariff or invoice. Observed on the v5.0.1 page; bound to Intelligence Index v5.0.1 and never compared with v5.0 rows (five of the stored v5.0 cost and token values differ on the v5.0.1 page). AA's label states no reasoning effort. Manual observation of Artificial Analysis's public Intelligence Index v5.0.1 evaluation page, "Cost per Intelligence Index Task" stacked chart (total label), 5 November 2026 (106 configurations selected with ?models=; rendered bar label paired with the x-axis model link; research-raw/aa-v5.0.1-2026-11-05/cap_index_cost.txt). Configuration "Inkling Small". AA's displayed rounding; no API, download, hidden precision or inferred run date.

37 k tokens/task · raw 37 k tokens/task

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Thousands of output tokens per task as displayed. Observed on the v5.0.1 page; bound to Intelligence Index v5.0.1 and never compared with v5.0 rows (five of the stored v5.0 cost and token values differ on the v5.0.1 page). AA's label states no reasoning effort. Manual observation of Artificial Analysis's public Intelligence Index v5.0.1 evaluation page, "Output Tokens per Task" stacked chart (reasoning plus answer total label), 5 November 2026 (106 configurations selected with ?models=; rendered bar label paired with the x-axis model link; research-raw/aa-v5.0.1-2026-11-05/cap_index_tokens.txt). Configuration "Inkling Small". AA's displayed rounding; no API, download, hidden precision or inferred run date.

18.5% · raw 18.5 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

CritPt v2 hard subset as regraded by AA on 3 November 2026; not the full set or the earlier CritPt (AA) rows. AA's label states no reasoning effort. Manual observation of Artificial Analysis's public CritPt v2 evaluation page, "Hard Subset" view, after AA's 3 November 2026 regrade, 5 November 2026 (106 configurations selected with ?models=; rendered bar label paired with the x-axis model link; research-raw/aa-v5.0.1-2026-11-05/cap_critpt_v2_hard.txt). Configuration "Inkling Small". AA's displayed rounding; no API, download, hidden precision or inferred run date.

4.8% · raw 4.8 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

AA's label states no reasoning effort. Manual observation of Artificial Analysis's public AutomationBench-AA evaluation page, Score chart, 5 November 2026 (106 configurations selected with ?models=; rendered bar label paired with the x-axis model link; research-raw/aa-v5.0.1-2026-11-05/cap_automationbench.txt). Configuration "Inkling Small". AA's displayed rounding; no API, download, hidden precision or inferred run date.

913 · raw 913 Elo

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

AA's label states no reasoning effort. Manual observation of Artificial Analysis's public AA-Briefcase evaluation page, combined Elo chart, 5 November 2026 (106 configurations selected with ?models=; rendered bar label paired with the x-axis model link; research-raw/aa-v5.0.1-2026-11-05/cap_briefcase_elo.txt). Configuration "Inkling Small". AA's displayed rounding; no API, download, hidden precision or inferred run date.

785 · raw 785 Elo

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

AA's label states no reasoning effort. Manual observation of Artificial Analysis's public AA-Briefcase evaluation page, "Analytical Quality & Presentation Elo" tab (Analytical Quality series), 5 November 2026 (106 configurations selected with ?models=; rendered bar label paired with the x-axis model link; research-raw/aa-v5.0.1-2026-11-05/cap_briefcase_quality_presentation.txt). Configuration "Inkling Small". AA's displayed rounding; no API, download, hidden precision or inferred run date.

1068 · raw 1068 Elo

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

AA's label states no reasoning effort. Manual observation of Artificial Analysis's public AA-Briefcase evaluation page, "Analytical Quality & Presentation Elo" tab (Presentation series), 5 November 2026 (106 configurations selected with ?models=; rendered bar label paired with the x-axis model link; research-raw/aa-v5.0.1-2026-11-05/cap_briefcase_quality_presentation.txt). Configuration "Inkling Small". AA's displayed rounding; no API, download, hidden precision or inferred run date.

20.0% · raw 20 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

AA's label states no reasoning effort. Manual observation of Artificial Analysis's public AA-Briefcase evaluation page, "Rubric Score (%)" tab, 5 November 2026 (106 configurations selected with ?models=; rendered bar label paired with the x-axis model link; research-raw/aa-v5.0.1-2026-11-05/cap_briefcase_rubric.txt). Configuration "Inkling Small". AA's displayed rounding; no API, download, hidden precision or inferred run date.

-9 · raw -9 index

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Signed knowledge index; v1.1 (3,000 questions) is not comparable with v1.0 rows. AA's label states no reasoning effort. Manual observation of Artificial Analysis's public AA-Omniscience evaluation page, "AA-Omniscience v1.1: Net Score" chart, 5 November 2026 (106 configurations selected with ?models=; rendered bar label paired with the x-axis model link; research-raw/aa-v5.0.1-2026-11-05/cap_omniscience.txt). Configuration "Inkling Small". AA's displayed rounding; no API, download, hidden precision or inferred run date.

32% · raw 32 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Not comparable with v1.0 rows. AA's label states no reasoning effort. Manual observation of Artificial Analysis's public AA-Omniscience evaluation page, accuracy chart, 5 November 2026 (106 configurations selected with ?models=; rendered bar label paired with the x-axis model link; research-raw/aa-v5.0.1-2026-11-05/cap_omniscience.txt). Configuration "Inkling Small". AA's displayed rounding; no API, download, hidden precision or inferred run date.

61% · raw 61 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Lower is better; not comparable with v1.0 rows. AA's label states no reasoning effort. Manual observation of Artificial Analysis's public AA-Omniscience evaluation page, hallucination rate chart, 5 November 2026 (106 configurations selected with ?models=; rendered bar label paired with the x-axis model link; research-raw/aa-v5.0.1-2026-11-05/cap_omniscience.txt). Configuration "Inkling Small". AA's displayed rounding; no API, download, hidden precision or inferred run date.

0% · raw 0 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

AA's label states no reasoning effort. Manual observation of Artificial Analysis's public Terminal-Bench-Science 0.1 evaluation page, Score chart, 5 November 2026 (106 configurations selected with ?models=; rendered bar label paired with the x-axis model link; research-raw/aa-v5.0.1-2026-11-05/cap_tbscience.txt). Configuration "Inkling Small". AA's displayed rounding; no API, download, hidden precision or inferred run date.

4.8% · raw 4.8 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

AA harness; not Cognition board rows. AA's label states no reasoning effort. Manual observation of Artificial Analysis's public FrontierCode 2.0 evaluation page, Score chart, 5 November 2026 (106 configurations selected with ?models=; rendered bar label paired with the x-axis model link; research-raw/aa-v5.0.1-2026-11-05/cap_frontiercode2.txt). Configuration "Inkling Small". AA's displayed rounding; no API, download, hidden precision or inferred run date.

12.1% · raw 12.1 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Not Humanity's Last Exam (AA). AA's label states no reasoning effort. Manual observation of Artificial Analysis's public HLE-Diamond evaluation page, Score chart, 5 November 2026 (106 configurations selected with ?models=; rendered bar label paired with the x-axis model link; research-raw/aa-v5.0.1-2026-11-05/cap_hle_diamond.txt). Configuration "Inkling Small". AA's displayed rounding; no API, download, hidden precision or inferred run date.

11.2% · raw 11.2 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

AA's label states no reasoning effort. Manual observation of Artificial Analysis's public GDP.pdf evaluation page, "All-pass Rate" tab, 5 November 2026 (106 configurations selected with ?models=; rendered bar label paired with the x-axis model link; research-raw/aa-v5.0.1-2026-11-05/cap_gdppdf.txt). Configuration "Inkling Small". AA's displayed rounding; no API, download, hidden precision or inferred run date.

1% · raw 1 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

AA's label states no reasoning effort. Manual observation of Artificial Analysis's public Terminal-Bench 4.0 evaluation page, Score chart, 5 November 2026 (106 configurations selected with ?models=; rendered bar label paired with the x-axis model link; research-raw/aa-v5.0.1-2026-11-05/cap_tb4.txt). Configuration "Inkling Small". AA's displayed rounding; no API, download, hidden precision or inferred run date.

151 tok/s · raw 151 tok/s

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

AA Performance Benchmarking v3.0.0 (3 November 2026): output tokens, including reasoning tokens, per second over the whole request including time to first token, averaged over the past 5 days for each workload (single-request and multi-turn agentic) and then across workloads; the model's first-party API, or the median across providers where there is none. A rolling observation; not the earlier "Output speed" rows. AA's label states no reasoning effort. Manual observation of Artificial Analysis's public Muse Spark 1.3 model page, Performance section "Output Speed" chart (Performance Benchmarking v3.0.0; the same ?models= selection), 5 November 2026 (106 configurations selected with ?models=; rendered bar label paired with the x-axis model link; research-raw/aa-v5.0.1-2026-11-05/cap_output_speed_v3.txt). Configuration "Inkling Small". AA's displayed rounding; no API, download, hidden precision or inferred run date.

27.590% · raw 27.59 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Vals exact model page https://www.vals.ai/models/thinkingmachines_inkling-small; source label Vals Index, 27.59%, published uncertainty ±1.13 percentage points; rank 33/33. Model page defaults are not inferred vendor product limits. Run execution date unpublished. Vals hyperparameter panel: provider Thinking Machines, temperature 1, top-p 1, max output 262,000 tokens, reasoning effort “0.99”, stored as effort unknown.

36.270% · raw 36.27 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Vals exact model page https://www.vals.ai/models/thinkingmachines_inkling-small; source label Vals Finance Index, 36.27%, published uncertainty ±1.56 percentage points; rank 36/36. Model page defaults are not inferred vendor product limits. Run execution date unpublished. Vals hyperparameter panel: provider Thinking Machines, temperature 1, top-p 1, max output 262,000 tokens, reasoning effort “0.99”, stored as effort unknown.

57.830% · raw 57.83 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Vals exact model page https://www.vals.ai/models/thinkingmachines_inkling-small; source label Vals Legal Index, 57.83%, published uncertainty ±1.70 percentage points; rank 32/34. Model page defaults are not inferred vendor product limits. Run execution date unpublished. Vals hyperparameter panel: provider Thinking Machines, temperature 1, top-p 1, max output 262,000 tokens, reasoning effort “0.99”, stored as effort unknown.

10.430% · raw 10.43 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Vals exact model page https://www.vals.ai/models/thinkingmachines_inkling-small; source label Code Migration v1.1, 10.43%, published uncertainty ±2.32 percentage points; rank 28/29. Model page defaults are not inferred vendor product limits. Run execution date unpublished. Vals hyperparameter panel: provider Thinking Machines, temperature 1, top-p 1, max output 262,000 tokens, reasoning effort “0.99”, stored as effort unknown.

64.1% · raw 64.11 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Vals exact model page https://www.vals.ai/models/thinkingmachines_inkling-small; source label CyberBench v1.1, 64.11%, published uncertainty ±6.17 percentage points; rank 34/47. Model page defaults are not inferred vendor product limits. Run execution date unpublished. Vals hyperparameter panel: provider Thinking Machines, temperature 1, top-p 1, max output 262,000 tokens, reasoning effort “0.99”, stored as effort unknown.

40.820% · raw 40.82 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Vals exact model page https://www.vals.ai/models/thinkingmachines_inkling-small; source label EMB v1.1, 40.82%, published uncertainty ±3.21 percentage points; rank 38/38. Model page defaults are not inferred vendor product limits. Run execution date unpublished. Vals hyperparameter panel: provider Thinking Machines, temperature 1, top-p 1, max output 262,000 tokens, reasoning effort “0.99”, stored as effort unknown.

44.780% · raw 44.78 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Vals exact model page https://www.vals.ai/models/thinkingmachines_inkling-small; source label Finance Agent v2.1, 44.78%, published uncertainty ±2.10 percentage points; rank 36/36. Model page defaults are not inferred vendor product limits. Run execution date unpublished. Vals hyperparameter panel: provider Thinking Machines, temperature 1, top-p 1, max output 262,000 tokens, reasoning effort “0.99”, stored as effort unknown.

25.480% · raw 25.48 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Vals exact model page https://www.vals.ai/models/thinkingmachines_inkling-small; source label Legal Research Bench, 25.48%, published uncertainty ±3.03 percentage points; rank 62/90. Model page defaults are not inferred vendor product limits. Run execution date unpublished. Vals hyperparameter panel: provider Thinking Machines, temperature 1, top-p 1, max output 262,000 tokens, reasoning effort “0.99”, stored as effort unknown.

84.1% · raw 84.11 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Vals exact model page https://www.vals.ai/models/thinkingmachines_inkling-small; source label MedScribe, 84.11%, published uncertainty ±1.87 percentage points; rank 38/109. Model page defaults are not inferred vendor product limits. Run execution date unpublished. Vals hyperparameter panel: provider Thinking Machines, temperature 1, top-p 1, max output 262,000 tokens, reasoning effort “0.99”, stored as effort unknown.

62.3% · raw 62.28 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Vals exact model page https://www.vals.ai/models/thinkingmachines_inkling-small; source label MortgageTax, 62.28%, published uncertainty ±0.95 percentage points; rank 55/98. Model page defaults are not inferred vendor product limits. Run execution date unpublished. Vals hyperparameter panel: provider Thinking Machines, temperature 1, top-p 1, max output 262,000 tokens, reasoning effort “0.99”, stored as effort unknown.

6% · raw 6 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Vals exact model page https://www.vals.ai/models/thinkingmachines_inkling-small; source label ProofBench v1.1, 6.00%, published uncertainty ±2.39 percentage points; rank 49/52. Model page defaults are not inferred vendor product limits. Run execution date unpublished. Vals hyperparameter panel: provider Thinking Machines, temperature 1, top-p 1, max output 262,000 tokens, reasoning effort “0.99”, stored as effort unknown.

59.4% · raw 59.41 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Vals exact model page https://www.vals.ai/models/thinkingmachines_inkling-small; source label Public Benefits Bench v1.1, 59.41%, published uncertainty ±1.28 percentage points; rank 34/50. Model page defaults are not inferred vendor product limits. Run execution date unpublished. Vals hyperparameter panel: provider Thinking Machines, temperature 1, top-p 1, max output 262,000 tokens, reasoning effort “0.99”, stored as effort unknown.

15.390% · raw 15.39 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Vals exact model page https://www.vals.ai/models/thinkingmachines_inkling-small; source label Tax Agent Bench, 15.39%, published uncertainty ±2.37 percentage points; rank 59/81. Model page defaults are not inferred vendor product limits. Run execution date unpublished. Vals hyperparameter panel: provider Thinking Machines, temperature 1, top-p 1, max output 262,000 tokens, reasoning effort “0.99”, stored as effort unknown.

19.050% · raw 19.05 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Vals exact model page https://www.vals.ai/models/thinkingmachines_inkling-small; source label Vibe Code Bench v1.1, 19.05%, published uncertainty ±3.40 percentage points; rank 95/124. Model page defaults are not inferred vendor product limits. Run execution date unpublished. Harness: OpenHands. Vals hyperparameter panel: provider Thinking Machines, temperature 1, top-p 1, max output 262,000 tokens, reasoning effort “0.99”, stored as effort unknown.

2.920% · raw 2.92 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Vals exact model page https://www.vals.ai/models/thinkingmachines_inkling-small; source label HLAB-Vals-1.1, 2.92%, published uncertainty ±1.36 percentage points; rank 33/34. Model page defaults are not inferred vendor product limits. Run execution date unpublished. Vals hyperparameter panel: provider Thinking Machines, temperature 1, top-p 1, max output 262,000 tokens, reasoning effort “0.99”, stored as effort unknown.

9.3% · raw 9.33 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Vals exact model page https://www.vals.ai/models/thinkingmachines_inkling-small; source label IOI, 9.33%, published uncertainty ±4.47 percentage points; rank 45/46. Model page defaults are not inferred vendor product limits. Run execution date unpublished. Vals hyperparameter panel: provider Thinking Machines, temperature 1, top-p 1, max output 262,000 tokens, reasoning effort “0.99”, stored as effort unknown.

0.5% · raw 0.5 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Vals exact model page https://www.vals.ai/models/thinkingmachines_inkling-small; source label ProgramBench, 0.50%, published uncertainty ±0.50 percentage points; rank 31/64. Model page defaults are not inferred vendor product limits. Run execution date unpublished. Vals hyperparameter panel: provider Thinking Machines, temperature 1, top-p 1, max output 262,000 tokens, reasoning effort “0.99”, stored as effort unknown.

82.2% · raw 82.2 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Vals exact model page https://www.vals.ai/models/thinkingmachines_inkling-small; source label SWE-bench, 82.20%, published uncertainty ±1.71 percentage points; rank 20/88. Model page defaults are not inferred vendor product limits. Run execution date unpublished. Vals hyperparameter panel: provider Thinking Machines, temperature 1, top-p 1, max output 262,000 tokens, reasoning effort “0.99”, stored as effort unknown.

0.000% · raw 0 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Vals exact model page https://www.vals.ai/models/thinkingmachines_inkling-small; source label Terminal-Bench 4.0, 0.00%, published uncertainty ±0.00 percentage points; rank 58/60. Model page defaults are not inferred vendor product limits. Run execution date unpublished. Harness: Mini-SWE-agent. Vals hyperparameter panel: provider Thinking Machines, temperature 1, top-p 1, max output 262,000 tokens, reasoning effort “0.99”, stored as effort unknown.

$0.401/task · raw 0.401 USD/task

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Publisher-displayed Vals Index mean cost per test on the exact model page; named Vals aggregate/harness only, not token pricing, subscription cost or provider entitlement. Suite run execution dates unpublished. Vals hyperparameter panel: provider Thinking Machines, temperature 1, top-p 1, max output 262,000 tokens, reasoning effort “0.99”, stored as effort unknown.

84% · raw 84 %

Headline · xhigh effort · Independent evaluator

Source/record date: 2026-11-05

Harness: Verified Semi-Private. Exact ARC-AGI-1 Semi-Private source configuration thinky-inkling-small-xhigh. Raw source fraction ×100; source artifact generation is not an execution date. Best verified score is headline, source matrix effort order breaks ties. Published cost is workload evidence, not token tariff.

78% · raw 78 %

Alternative · high effort · Independent evaluator

Source/record date: 2026-11-05

Harness: Verified Semi-Private. Exact ARC-AGI-1 Semi-Private source configuration thinky-inkling-small-high. Raw source fraction ×100; source artifact generation is not an execution date. Best verified score is headline, source matrix effort order breaks ties. Published cost is workload evidence, not token tariff.

67% · raw 67 %

Alternative · medium effort · Independent evaluator

Source/record date: 2026-11-05

Harness: Verified Semi-Private. Exact ARC-AGI-1 Semi-Private source configuration thinky-inkling-small-medium. Raw source fraction ×100; source artifact generation is not an execution date. Best verified score is headline, source matrix effort order breaks ties. Published cost is workload evidence, not token tariff.

52.5% · raw 52.5 %

Alternative · low effort · Independent evaluator

Source/record date: 2026-11-05

Harness: Verified Semi-Private. Exact ARC-AGI-1 Semi-Private source configuration thinky-inkling-small-low. Raw source fraction ×100; source artifact generation is not an execution date. Best verified score is headline, source matrix effort order breaks ties. Published cost is workload evidence, not token tariff.

48.5% · raw 48.5 %

Alternative · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: Verified Semi-Private. Exact ARC-AGI-1 Semi-Private source configuration thinky-inkling-small-minimal. Raw source fraction ×100; source artifact generation is not an execution date. Best verified score is headline, source matrix effort order breaks ties. Published cost is workload evidence, not token tariff. Source effort label “minimal” is not one of the named levels used here, so effort stays unknown.

5.5% · raw 5.5 %

Alternative · none effort · Independent evaluator

Source/record date: 2026-11-05

Harness: Verified Semi-Private. Exact ARC-AGI-1 Semi-Private source configuration thinky-inkling-small-none. Raw source fraction ×100; source artifact generation is not an execution date. Best verified score is headline, source matrix effort order breaks ties. Published cost is workload evidence, not token tariff.

40.1% · raw 40.138888888888886 %

Headline · xhigh effort · Independent evaluator

Source/record date: 2026-11-05

Harness: Verified Semi-Private. Exact ARC-AGI-2 Semi-Private source configuration thinky-inkling-small-xhigh. Raw source fraction ×100; source artifact generation is not an execution date. Best verified score is headline, source matrix effort order breaks ties. Published cost is workload evidence, not token tariff.

33.1% · raw 33.05555555555556 %

Alternative · high effort · Independent evaluator

Source/record date: 2026-11-05

Harness: Verified Semi-Private. Exact ARC-AGI-2 Semi-Private source configuration thinky-inkling-small-high. Raw source fraction ×100; source artifact generation is not an execution date. Best verified score is headline, source matrix effort order breaks ties. Published cost is workload evidence, not token tariff.

13.6% · raw 13.611111111111112 %

Alternative · medium effort · Independent evaluator

Source/record date: 2026-11-05

Harness: Verified Semi-Private. Exact ARC-AGI-2 Semi-Private source configuration thinky-inkling-small-medium. Raw source fraction ×100; source artifact generation is not an execution date. Best verified score is headline, source matrix effort order breaks ties. Published cost is workload evidence, not token tariff.

4.9% · raw 4.861111111111112 %

Alternative · low effort · Independent evaluator

Source/record date: 2026-11-05

Harness: Verified Semi-Private. Exact ARC-AGI-2 Semi-Private source configuration thinky-inkling-small-low. Raw source fraction ×100; source artifact generation is not an execution date. Best verified score is headline, source matrix effort order breaks ties. Published cost is workload evidence, not token tariff.

4% · raw 4.027777777777778 %

Alternative · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: Verified Semi-Private. Exact ARC-AGI-2 Semi-Private source configuration thinky-inkling-small-minimal. Raw source fraction ×100; source artifact generation is not an execution date. Best verified score is headline, source matrix effort order breaks ties. Published cost is workload evidence, not token tariff. Source effort label “minimal” is not one of the named levels used here, so effort stays unknown.

0% · raw 0 %

Alternative · none effort · Independent evaluator

Source/record date: 2026-11-05

Harness: Verified Semi-Private. Exact ARC-AGI-2 Semi-Private source configuration thinky-inkling-small-none. Raw source fraction ×100; source artifact generation is not an execution date. Best verified score is headline, source matrix effort order breaks ties. Published cost is workload evidence, not token tariff.

79.2% · raw 79.2 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: Scale MCP Atlas updated April 2026 method, maximum 100 tool calls/task, transient-error retry, updated LLM judge. Exact visible Inkling-Small pass rate 79.20%, 95% interval half-width 2.50 percentage points. Dataset 1000 tasks (500 public/500 heldout); page describes the public leaderboard subset as 500 tasks. 36 real MCP servers/220 tools. Footer update date does not establish this exact model execution date. No effort label shown.

-0.170 IPS score · raw -0.17010177604397714 IPS score

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-01

Harness: Arena Agent Arena sessions with randomized component selection. Inverse-propensity-weighted estimate of the causal effect of this model choice on the aggregate of its user-outcome signals, relative to a uniform baseline over components, with time-decaying weights. Signed and relative to the published pool; zero is that baseline, not zero capability. 95% confidence interval -0.1884 to -0.1518; 552,386 observations; 14,622 sessions; rank 56 of 57 including unregistered models. Board published 28 October 2026 (dataset revision 323062f7, captured 1 November); a newer board may exist. Individual run dates and checkpoint hashes unpublished. No effort label on the board.

-0.281 IPS score · raw -0.2814618296709025 IPS score

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-01

Harness: Arena Agent Arena sessions with randomized component selection. Inverse-propensity-weighted estimate of the causal effect of this model choice on explicit user confirmation that the task was completed, relative to a uniform baseline over components, with time-decaying weights. Signed and relative to the published pool; zero is that baseline, not zero capability. 95% confidence interval -0.3293 to -0.2336; 6,995 observations; unpublished sessions; rank 56 of 57 including unregistered models. Board published 28 October 2026 (dataset revision 323062f7, captured 1 November); a newer board may exist. Individual run dates and checkpoint hashes unpublished. No effort label on the board.

-0.227 IPS score · raw -0.22687753908282782 IPS score

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-01

Harness: Arena Agent Arena sessions with randomized component selection. Inverse-propensity-weighted estimate of the causal effect of this model choice on tasks with more unprompted praise than complaints, relative to a uniform baseline over components, with time-decaying weights. Signed and relative to the published pool; zero is that baseline, not zero capability. 95% confidence interval -0.2620 to -0.1918; 3,425 observations; unpublished sessions; rank 54 of 57 including unregistered models. Board published 28 October 2026 (dataset revision 323062f7, captured 1 November); a newer board may exist. Individual run dates and checkpoint hashes unpublished. No effort label on the board.

-0.248 IPS score · raw -0.24795442603643925 IPS score

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-01

Harness: Arena Agent Arena sessions with randomized component selection. Inverse-propensity-weighted estimate of the causal effect of this model choice on less steering effort required from the user, relative to a uniform baseline over components, with time-decaying weights. Signed and relative to the published pool; zero is that baseline, not zero capability. 95% confidence interval -0.2839 to -0.2120; 36,006 observations; unpublished sessions; rank 57 of 57 including unregistered models. Board published 28 October 2026 (dataset revision 323062f7, captured 1 November); a newer board may exist. Individual run dates and checkpoint hashes unpublished. No effort label on the board.

0.035 IPS score · raw 0.03457660727740128 IPS score

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-01

Harness: Arena Agent Arena sessions with randomized component selection. Inverse-propensity-weighted estimate of the causal effect of this model choice on fewer agent-caused bash failures per call, relative to a uniform baseline over components, with time-decaying weights. Signed and relative to the published pool; zero is that baseline, not zero capability. 95% confidence interval 0.0053 to 0.0638; 14,985 observations; unpublished sessions; rank 25 of 57 including unregistered models. Board published 28 October 2026 (dataset revision 323062f7, captured 1 November); a newer board may exist. Individual run dates and checkpoint hashes unpublished. No effort label on the board.

-0.026 IPS score · raw -0.026452501046428267 IPS score

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-01

Harness: Arena Agent Arena sessions with randomized component selection. Inverse-propensity-weighted estimate of the causal effect of this model choice on fewer hallucinated tool calls, relative to a uniform baseline over components, with time-decaying weights. Signed and relative to the published pool; zero is that baseline, not zero capability. 95% confidence interval -0.0419 to -0.0110; 490,975 observations; unpublished sessions; rank 50 of 57 including unregistered models. Board published 28 October 2026 (dataset revision 323062f7, captured 1 November); a newer board may exist. Individual run dates and checkpoint hashes unpublished. No effort label on the board.

$0.249301/task · raw 0.249301 USD/task

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: published Vals component protocol. Exact Vals task/version and source model. Requested effort, provider and diagnostics are retained with this observation; evaluation date and checkpoint hash are unpublished. Source update date is not execution date. Quality standard error is not a confidence interval. A source revision does not by itself establish a model update. Workload diagnostic; quality uncertainty does not describe its uncertainty. Not a token tariff or controlled efficiency ranking.

1173.625 s · raw 1173.625 s

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: published Vals component protocol. Exact Vals task/version and source model. Requested effort, provider and diagnostics are retained with this observation; evaluation date and checkpoint hash are unpublished. Source update date is not execution date. Quality standard error is not a confidence interval. A source revision does not by itself establish a model update. Workload diagnostic; quality uncertainty does not describe its uncertainty. Not a token tariff or controlled efficiency ranking.

37.824% · raw 37.824 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: published Vals component protocol. Exact Vals task/version and source model. Requested effort, provider and diagnostics are retained with this observation; evaluation date and checkpoint hash are unpublished. Source update date is not execution date. Quality standard error is not a confidence interval. A source revision does not by itself establish a model update.

43.535% · raw 43.535 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: published Vals component protocol. Exact Vals task/version and source model. Requested effort, provider and diagnostics are retained with this observation; evaluation date and checkpoint hash are unpublished. Source update date is not execution date. Quality standard error is not a confidence interval. A source revision does not by itself establish a model update.

26.761% · raw 26.761 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: published Vals component protocol. Exact Vals task/version and source model. Requested effort, provider and diagnostics are retained with this observation; evaluation date and checkpoint hash are unpublished. Source update date is not execution date. Quality standard error is not a confidence interval. A source revision does not by itself establish a model update.

34.665% · raw 34.665 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: published Vals component protocol. Exact Vals task/version and source model. Requested effort, provider and diagnostics are retained with this observation; evaluation date and checkpoint hash are unpublished. Source update date is not execution date. Quality standard error is not a confidence interval. A source revision does not by itself establish a model update.

42.095% · raw 42.095 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: published Vals component protocol. Exact Vals task/version and source model. Requested effort, provider and diagnostics are retained with this observation; evaluation date and checkpoint hash are unpublished. Source update date is not execution date. Quality standard error is not a confidence interval. A source revision does not by itself establish a model update.

40.964% · raw 40.964 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: published Vals component protocol. Exact Vals task/version and source model. Requested effort, provider and diagnostics are retained with this observation; evaluation date and checkpoint hash are unpublished. Source update date is not execution date. Quality standard error is not a confidence interval. A source revision does not by itself establish a model update.

45.630% · raw 45.63 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: published Vals component protocol. Exact Vals task/version and source model. Requested effort, provider and diagnostics are retained with this observation; evaluation date and checkpoint hash are unpublished. Source update date is not execution date. Quality standard error is not a confidence interval. A source revision does not by itself establish a model update.

56.620% · raw 56.62 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: published Vals component protocol. Exact Vals task/version and source model. Requested effort, provider and diagnostics are retained with this observation; evaluation date and checkpoint hash are unpublished. Source update date is not execution date. Quality standard error is not a confidence interval. A source revision does not by itself establish a model update.

39.029% · raw 39.029 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: published Vals component protocol. Exact Vals task/version and source model. Requested effort, provider and diagnostics are retained with this observation; evaluation date and checkpoint hash are unpublished. Source update date is not execution date. Quality standard error is not a confidence interval. A source revision does not by itself establish a model update.

$0.182996/task · raw 0.182996 USD/task

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: published Vals component protocol. Exact Vals task/version and source model. Requested effort, provider and diagnostics are retained with this observation; evaluation date and checkpoint hash are unpublished. Source update date is not execution date. Quality standard error is not a confidence interval. A source revision does not by itself establish a model update. Workload diagnostic; quality uncertainty does not describe its uncertainty. Not a token tariff or controlled efficiency ranking.

624.364 s · raw 624.364 s

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: published Vals component protocol. Exact Vals task/version and source model. Requested effort, provider and diagnostics are retained with this observation; evaluation date and checkpoint hash are unpublished. Source update date is not execution date. Quality standard error is not a confidence interval. A source revision does not by itself establish a model update. Workload diagnostic; quality uncertainty does not describe its uncertainty. Not a token tariff or controlled efficiency ranking.

44.776% · raw 44.776 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: published Vals component protocol. Exact Vals task/version and source model. Requested effort, provider and diagnostics are retained with this observation; evaluation date and checkpoint hash are unpublished. Source update date is not execution date. Quality standard error is not a confidence interval. A source revision does not by itself establish a model update.

40.823% · raw 40.823 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: published Vals component protocol. Exact Vals task/version and source model. Requested effort, provider and diagnostics are retained with this observation; evaluation date and checkpoint hash are unpublished. Source update date is not execution date. Quality standard error is not a confidence interval. A source revision does not by itself establish a model update.

15.389% · raw 15.389 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: published Vals component protocol. Exact Vals task/version and source model. Requested effort, provider and diagnostics are retained with this observation; evaluation date and checkpoint hash are unpublished. Source update date is not execution date. Quality standard error is not a confidence interval. A source revision does not by itself establish a model update.

$0.207921/task · raw 0.207921 USD/task

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: published Vals component protocol. Exact Vals task/version and source model. Requested effort, provider and diagnostics are retained with this observation; evaluation date and checkpoint hash are unpublished. Source update date is not execution date. Quality standard error is not a confidence interval. A source revision does not by itself establish a model update. Workload diagnostic; quality uncertainty does not describe its uncertainty. Not a token tariff or controlled efficiency ranking.

314.216 s · raw 314.216 s

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: published Vals component protocol. Exact Vals task/version and source model. Requested effort, provider and diagnostics are retained with this observation; evaluation date and checkpoint hash are unpublished. Source update date is not execution date. Quality standard error is not a confidence interval. A source revision does not by itself establish a model update. Workload diagnostic; quality uncertainty does not describe its uncertainty. Not a token tariff or controlled efficiency ranking.

31.855% · raw 31.855 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: published Vals component protocol. Exact Vals task/version and source model. Requested effort, provider and diagnostics are retained with this observation; evaluation date and checkpoint hash are unpublished. Source update date is not execution date. Quality standard error is not a confidence interval. A source revision does not by itself establish a model update.

57.736% · raw 57.736 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: published Vals component protocol. Exact Vals task/version and source model. Requested effort, provider and diagnostics are retained with this observation; evaluation date and checkpoint hash are unpublished. Source update date is not execution date. Quality standard error is not a confidence interval. A source revision does not by itself establish a model update.

61.019% · raw 61.019 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: published Vals component protocol. Exact Vals task/version and source model. Requested effort, provider and diagnostics are retained with this observation; evaluation date and checkpoint hash are unpublished. Source update date is not execution date. Quality standard error is not a confidence interval. A source revision does not by itself establish a model update.

43.645% · raw 43.645 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: published Vals component protocol. Exact Vals task/version and source model. Requested effort, provider and diagnostics are retained with this observation; evaluation date and checkpoint hash are unpublished. Source update date is not execution date. Quality standard error is not a confidence interval. A source revision does not by itself establish a model update.

42.485% · raw 42.485 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: published Vals component protocol. Exact Vals task/version and source model. Requested effort, provider and diagnostics are retained with this observation; evaluation date and checkpoint hash are unpublished. Source update date is not execution date. Quality standard error is not a confidence interval. A source revision does not by itself establish a model update.

14.159% · raw 14.159 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: published Vals component protocol. Exact Vals task/version and source model. Requested effort, provider and diagnostics are retained with this observation; evaluation date and checkpoint hash are unpublished. Source update date is not execution date. Quality standard error is not a confidence interval. A source revision does not by itself establish a model update.

39.373% · raw 39.373 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: published Vals component protocol. Exact Vals task/version and source model. Requested effort, provider and diagnostics are retained with this observation; evaluation date and checkpoint hash are unpublished. Source update date is not execution date. Quality standard error is not a confidence interval. A source revision does not by itself establish a model update.

64.947% · raw 64.947 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: published Vals component protocol. Exact Vals task/version and source model. Requested effort, provider and diagnostics are retained with this observation; evaluation date and checkpoint hash are unpublished. Source update date is not execution date. Quality standard error is not a confidence interval. A source revision does not by itself establish a model update.

54.622% · raw 54.622 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: published Vals component protocol. Exact Vals task/version and source model. Requested effort, provider and diagnostics are retained with this observation; evaluation date and checkpoint hash are unpublished. Source update date is not execution date. Quality standard error is not a confidence interval. A source revision does not by itself establish a model update.

24.996% · raw 24.996 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: published Vals component protocol. Exact Vals task/version and source model. Requested effort, provider and diagnostics are retained with this observation; evaluation date and checkpoint hash are unpublished. Source update date is not execution date. Quality standard error is not a confidence interval. A source revision does not by itself establish a model update.

$0.17869/task · raw 0.17869 USD/task

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: published Vals component protocol. Exact Vals task/version and source model. Requested effort, provider and diagnostics are retained with this observation; evaluation date and checkpoint hash are unpublished. Source update date is not execution date. Quality standard error is not a confidence interval. A source revision does not by itself establish a model update. Workload diagnostic; quality uncertainty does not describe its uncertainty. Not a token tariff or controlled efficiency ranking.

455.265 s · raw 455.265 s

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: published Vals component protocol. Exact Vals task/version and source model. Requested effort, provider and diagnostics are retained with this observation; evaluation date and checkpoint hash are unpublished. Source update date is not execution date. Quality standard error is not a confidence interval. A source revision does not by itself establish a model update. Workload diagnostic; quality uncertainty does not describe its uncertainty. Not a token tariff or controlled efficiency ranking.

0.000% · raw 0 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: published Vals component protocol. Exact Vals task/version and source model. Requested effort, provider and diagnostics are retained with this observation; evaluation date and checkpoint hash are unpublished. Source update date is not execution date. Quality standard error is not a confidence interval. A source revision does not by itself establish a model update.

0.000% · raw 0 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: published Vals component protocol. Exact Vals task/version and source model. Requested effort, provider and diagnostics are retained with this observation; evaluation date and checkpoint hash are unpublished. Source update date is not execution date. Quality standard error is not a confidence interval. A source revision does not by itself establish a model update.

0.000% · raw 0 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: published Vals component protocol. Exact Vals task/version and source model. Requested effort, provider and diagnostics are retained with this observation; evaluation date and checkpoint hash are unpublished. Source update date is not execution date. Quality standard error is not a confidence interval. A source revision does not by itself establish a model update.

0.000% · raw 0 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: published Vals component protocol. Exact Vals task/version and source model. Requested effort, provider and diagnostics are retained with this observation; evaluation date and checkpoint hash are unpublished. Source update date is not execution date. Quality standard error is not a confidence interval. A source revision does not by itself establish a model update.

20.000% · raw 20 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: published Vals component protocol. Exact Vals task/version and source model. Requested effort, provider and diagnostics are retained with this observation; evaluation date and checkpoint hash are unpublished. Source update date is not execution date. Quality standard error is not a confidence interval. A source revision does not by itself establish a model update.

0.000% · raw 0 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: published Vals component protocol. Exact Vals task/version and source model. Requested effort, provider and diagnostics are retained with this observation; evaluation date and checkpoint hash are unpublished. Source update date is not execution date. Quality standard error is not a confidence interval. A source revision does not by itself establish a model update.

20.000% · raw 20 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: published Vals component protocol. Exact Vals task/version and source model. Requested effort, provider and diagnostics are retained with this observation; evaluation date and checkpoint hash are unpublished. Source update date is not execution date. Quality standard error is not a confidence interval. A source revision does not by itself establish a model update.

0.000% · raw 0 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: published Vals component protocol. Exact Vals task/version and source model. Requested effort, provider and diagnostics are retained with this observation; evaluation date and checkpoint hash are unpublished. Source update date is not execution date. Quality standard error is not a confidence interval. A source revision does not by itself establish a model update.

0.000% · raw 0 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: published Vals component protocol. Exact Vals task/version and source model. Requested effort, provider and diagnostics are retained with this observation; evaluation date and checkpoint hash are unpublished. Source update date is not execution date. Quality standard error is not a confidence interval. A source revision does not by itself establish a model update.

0.000% · raw 0 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: published Vals component protocol. Exact Vals task/version and source model. Requested effort, provider and diagnostics are retained with this observation; evaluation date and checkpoint hash are unpublished. Source update date is not execution date. Quality standard error is not a confidence interval. A source revision does not by itself establish a model update.

10.000% · raw 10 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: published Vals component protocol. Exact Vals task/version and source model. Requested effort, provider and diagnostics are retained with this observation; evaluation date and checkpoint hash are unpublished. Source update date is not execution date. Quality standard error is not a confidence interval. A source revision does not by itself establish a model update.

0.000% · raw 0 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: published Vals component protocol. Exact Vals task/version and source model. Requested effort, provider and diagnostics are retained with this observation; evaluation date and checkpoint hash are unpublished. Source update date is not execution date. Quality standard error is not a confidence interval. A source revision does not by itself establish a model update.

0.000% · raw 0 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: published Vals component protocol. Exact Vals task/version and source model. Requested effort, provider and diagnostics are retained with this observation; evaluation date and checkpoint hash are unpublished. Source update date is not execution date. Quality standard error is not a confidence interval. A source revision does not by itself establish a model update.

10.000% · raw 10 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: published Vals component protocol. Exact Vals task/version and source model. Requested effort, provider and diagnostics are retained with this observation; evaluation date and checkpoint hash are unpublished. Source update date is not execution date. Quality standard error is not a confidence interval. A source revision does not by itself establish a model update.

0.000% · raw 0 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: published Vals component protocol. Exact Vals task/version and source model. Requested effort, provider and diagnostics are retained with this observation; evaluation date and checkpoint hash are unpublished. Source update date is not execution date. Quality standard error is not a confidence interval. A source revision does not by itself establish a model update.

0.000% · raw 0 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: published Vals component protocol. Exact Vals task/version and source model. Requested effort, provider and diagnostics are retained with this observation; evaluation date and checkpoint hash are unpublished. Source update date is not execution date. Quality standard error is not a confidence interval. A source revision does not by itself establish a model update.

0.000% · raw 0 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: published Vals component protocol. Exact Vals task/version and source model. Requested effort, provider and diagnostics are retained with this observation; evaluation date and checkpoint hash are unpublished. Source update date is not execution date. Quality standard error is not a confidence interval. A source revision does not by itself establish a model update.

0.000% · raw 0 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: published Vals component protocol. Exact Vals task/version and source model. Requested effort, provider and diagnostics are retained with this observation; evaluation date and checkpoint hash are unpublished. Source update date is not execution date. Quality standard error is not a confidence interval. A source revision does not by itself establish a model update.

0.000% · raw 0 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: published Vals component protocol. Exact Vals task/version and source model. Requested effort, provider and diagnostics are retained with this observation; evaluation date and checkpoint hash are unpublished. Source update date is not execution date. Quality standard error is not a confidence interval. A source revision does not by itself establish a model update.

0.000% · raw 0 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: published Vals component protocol. Exact Vals task/version and source model. Requested effort, provider and diagnostics are retained with this observation; evaluation date and checkpoint hash are unpublished. Source update date is not execution date. Quality standard error is not a confidence interval. A source revision does not by itself establish a model update.

0.000% · raw 0 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: published Vals component protocol. Exact Vals task/version and source model. Requested effort, provider and diagnostics are retained with this observation; evaluation date and checkpoint hash are unpublished. Source update date is not execution date. Quality standard error is not a confidence interval. A source revision does not by itself establish a model update.

0.000% · raw 0 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: published Vals component protocol. Exact Vals task/version and source model. Requested effort, provider and diagnostics are retained with this observation; evaluation date and checkpoint hash are unpublished. Source update date is not execution date. Quality standard error is not a confidence interval. A source revision does not by itself establish a model update.

10.000% · raw 10 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: published Vals component protocol. Exact Vals task/version and source model. Requested effort, provider and diagnostics are retained with this observation; evaluation date and checkpoint hash are unpublished. Source update date is not execution date. Quality standard error is not a confidence interval. A source revision does not by itself establish a model update.

0.000% · raw 0 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: published Vals component protocol. Exact Vals task/version and source model. Requested effort, provider and diagnostics are retained with this observation; evaluation date and checkpoint hash are unpublished. Source update date is not execution date. Quality standard error is not a confidence interval. A source revision does not by itself establish a model update.

81.493% · raw 81.493 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: published Vals component protocol. Exact Vals task/version and source model. Requested effort, provider and diagnostics are retained with this observation; evaluation date and checkpoint hash are unpublished. Source update date is not execution date. Quality standard error is not a confidence interval. A source revision does not by itself establish a model update.

81.522% · raw 81.522 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: published Vals component protocol. Exact Vals task/version and source model. Requested effort, provider and diagnostics are retained with this observation; evaluation date and checkpoint hash are unpublished. Source update date is not execution date. Quality standard error is not a confidence interval. A source revision does not by itself establish a model update.

75.269% · raw 75.269 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: published Vals component protocol. Exact Vals task/version and source model. Requested effort, provider and diagnostics are retained with this observation; evaluation date and checkpoint hash are unpublished. Source update date is not execution date. Quality standard error is not a confidence interval. A source revision does not by itself establish a model update.

88.433% · raw 88.433 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: published Vals component protocol. Exact Vals task/version and source model. Requested effort, provider and diagnostics are retained with this observation; evaluation date and checkpoint hash are unpublished. Source update date is not execution date. Quality standard error is not a confidence interval. A source revision does not by itself establish a model update.

75.779% · raw 75.779 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: published Vals component protocol. Exact Vals task/version and source model. Requested effort, provider and diagnostics are retained with this observation; evaluation date and checkpoint hash are unpublished. Source update date is not execution date. Quality standard error is not a confidence interval. A source revision does not by itself establish a model update.

86.471% · raw 86.471 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: published Vals component protocol. Exact Vals task/version and source model. Requested effort, provider and diagnostics are retained with this observation; evaluation date and checkpoint hash are unpublished. Source update date is not execution date. Quality standard error is not a confidence interval. A source revision does not by itself establish a model update.

87.500% · raw 87.5 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: published Vals component protocol. Exact Vals task/version and source model. Requested effort, provider and diagnostics are retained with this observation; evaluation date and checkpoint hash are unpublished. Source update date is not execution date. Quality standard error is not a confidence interval. A source revision does not by itself establish a model update.

92.444% · raw 92.444 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: published Vals component protocol. Exact Vals task/version and source model. Requested effort, provider and diagnostics are retained with this observation; evaluation date and checkpoint hash are unpublished. Source update date is not execution date. Quality standard error is not a confidence interval. A source revision does not by itself establish a model update.

84.806% · raw 84.806 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: published Vals component protocol. Exact Vals task/version and source model. Requested effort, provider and diagnostics are retained with this observation; evaluation date and checkpoint hash are unpublished. Source update date is not execution date. Quality standard error is not a confidence interval. A source revision does not by itself establish a model update.

74.744% · raw 74.744 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: published Vals component protocol. Exact Vals task/version and source model. Requested effort, provider and diagnostics are retained with this observation; evaluation date and checkpoint hash are unpublished. Source update date is not execution date. Quality standard error is not a confidence interval. A source revision does not by itself establish a model update.

77.903% · raw 77.903 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: published Vals component protocol. Exact Vals task/version and source model. Requested effort, provider and diagnostics are retained with this observation; evaluation date and checkpoint hash are unpublished. Source update date is not execution date. Quality standard error is not a confidence interval. A source revision does not by itself establish a model update.

79.868% · raw 79.868 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: published Vals component protocol. Exact Vals task/version and source model. Requested effort, provider and diagnostics are retained with this observation; evaluation date and checkpoint hash are unpublished. Source update date is not execution date. Quality standard error is not a confidence interval. A source revision does not by itself establish a model update.

76.165% · raw 76.165 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: published Vals component protocol. Exact Vals task/version and source model. Requested effort, provider and diagnostics are retained with this observation; evaluation date and checkpoint hash are unpublished. Source update date is not execution date. Quality standard error is not a confidence interval. A source revision does not by itself establish a model update.

80.466% · raw 80.466 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: published Vals component protocol. Exact Vals task/version and source model. Requested effort, provider and diagnostics are retained with this observation; evaluation date and checkpoint hash are unpublished. Source update date is not execution date. Quality standard error is not a confidence interval. A source revision does not by itself establish a model update.

90.772% · raw 90.772 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: published Vals component protocol. Exact Vals task/version and source model. Requested effort, provider and diagnostics are retained with this observation; evaluation date and checkpoint hash are unpublished. Source update date is not execution date. Quality standard error is not a confidence interval. A source revision does not by itself establish a model update.

81.273% · raw 81.273 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: published Vals component protocol. Exact Vals task/version and source model. Requested effort, provider and diagnostics are retained with this observation; evaluation date and checkpoint hash are unpublished. Source update date is not execution date. Quality standard error is not a confidence interval. A source revision does not by itself establish a model update.

82.904% · raw 82.904 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: published Vals component protocol. Exact Vals task/version and source model. Requested effort, provider and diagnostics are retained with this observation; evaluation date and checkpoint hash are unpublished. Source update date is not execution date. Quality standard error is not a confidence interval. A source revision does not by itself establish a model update.

86.655% · raw 86.655 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: published Vals component protocol. Exact Vals task/version and source model. Requested effort, provider and diagnostics are retained with this observation; evaluation date and checkpoint hash are unpublished. Source update date is not execution date. Quality standard error is not a confidence interval. A source revision does not by itself establish a model update.

85.458% · raw 85.458 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: published Vals component protocol. Exact Vals task/version and source model. Requested effort, provider and diagnostics are retained with this observation; evaluation date and checkpoint hash are unpublished. Source update date is not execution date. Quality standard error is not a confidence interval. A source revision does not by itself establish a model update.

80.639% · raw 80.639 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: published Vals component protocol. Exact Vals task/version and source model. Requested effort, provider and diagnostics are retained with this observation; evaluation date and checkpoint hash are unpublished. Source update date is not execution date. Quality standard error is not a confidence interval. A source revision does not by itself establish a model update.

77.551% · raw 77.551 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: published Vals component protocol. Exact Vals task/version and source model. Requested effort, provider and diagnostics are retained with this observation; evaluation date and checkpoint hash are unpublished. Source update date is not execution date. Quality standard error is not a confidence interval. A source revision does not by itself establish a model update.

77.097% · raw 77.097 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: published Vals component protocol. Exact Vals task/version and source model. Requested effort, provider and diagnostics are retained with this observation; evaluation date and checkpoint hash are unpublished. Source update date is not execution date. Quality standard error is not a confidence interval. A source revision does not by itself establish a model update.

80.435% · raw 80.435 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: published Vals component protocol. Exact Vals task/version and source model. Requested effort, provider and diagnostics are retained with this observation; evaluation date and checkpoint hash are unpublished. Source update date is not execution date. Quality standard error is not a confidence interval. A source revision does not by itself establish a model update.

81.489% · raw 81.489 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: published Vals component protocol. Exact Vals task/version and source model. Requested effort, provider and diagnostics are retained with this observation; evaluation date and checkpoint hash are unpublished. Source update date is not execution date. Quality standard error is not a confidence interval. A source revision does not by itself establish a model update.

71.886% · raw 71.886 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: published Vals component protocol. Exact Vals task/version and source model. Requested effort, provider and diagnostics are retained with this observation; evaluation date and checkpoint hash are unpublished. Source update date is not execution date. Quality standard error is not a confidence interval. A source revision does not by itself establish a model update.

986.441 s · raw 986.441 s

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: published Vals component protocol. Exact Vals task/version and source model. Requested effort, provider and diagnostics are retained with this observation; evaluation date and checkpoint hash are unpublished. Source update date is not execution date. Quality standard error is not a confidence interval. A source revision does not by itself establish a model update. Workload diagnostic; quality uncertainty does not describe its uncertainty. Not a token tariff or controlled efficiency ranking.

19.052% · raw 19.052 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: OpenHands. Exact Vals task/version and source model. Requested effort, provider and diagnostics are retained with this observation; evaluation date and checkpoint hash are unpublished. Source update date is not execution date. Quality standard error is not a confidence interval. A source revision does not by itself establish a model update.

44.776% · raw 44.776 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: published Vals component protocol. Exact Vals task/version and source model. Requested effort, provider and diagnostics are retained with this observation; evaluation date and checkpoint hash are unpublished. Source update date is not execution date. Quality standard error is not a confidence interval. A source revision does not by itself establish a model update.

40.823% · raw 40.823 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: published Vals component protocol. Exact Vals task/version and source model. Requested effort, provider and diagnostics are retained with this observation; evaluation date and checkpoint hash are unpublished. Source update date is not execution date. Quality standard error is not a confidence interval. A source revision does not by itself establish a model update.

10.398% · raw 10.398 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: published Vals component protocol. Exact Vals task/version and source model. Requested effort, provider and diagnostics are retained with this observation; evaluation date and checkpoint hash are unpublished. Source update date is not execution date. Quality standard error is not a confidence interval. A source revision does not by itself establish a model update. Index Code Migration is the 60-task subset (50 CLI + 10 COBOL), not standalone Migration.

25.481% · raw 25.481 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: published Vals component protocol. Exact Vals task/version and source model. Requested effort, provider and diagnostics are retained with this observation; evaluation date and checkpoint hash are unpublished. Source update date is not execution date. Quality standard error is not a confidence interval. A source revision does not by itself establish a model update.

2.917% · raw 2.917 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: published Vals component protocol. Exact Vals task/version and source model. Requested effort, provider and diagnostics are retained with this observation; evaluation date and checkpoint hash are unpublished. Source update date is not execution date. Quality standard error is not a confidence interval. A source revision does not by itself establish a model update.

0.000% · raw 0 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: published Vals component protocol. Exact Vals task/version and source model. Requested effort, provider and diagnostics are retained with this observation; evaluation date and checkpoint hash are unpublished. Source update date is not execution date. Quality standard error is not a confidence interval. A source revision does not by itself establish a model update.

15.389% · raw 15.389 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: published Vals component protocol. Exact Vals task/version and source model. Requested effort, provider and diagnostics are retained with this observation; evaluation date and checkpoint hash are unpublished. Source update date is not execution date. Quality standard error is not a confidence interval. A source revision does not by itself establish a model update.

$0.137335/task · raw 0.137335 USD/task

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: published Vals component protocol. Exact Vals task/version and source model. Requested effort, provider and diagnostics are retained with this observation; evaluation date and checkpoint hash are unpublished. Source update date is not execution date. Quality standard error is not a confidence interval. A source revision does not by itself establish a model update. Workload diagnostic; quality uncertainty does not describe its uncertainty. Not a token tariff or controlled efficiency ranking.

538.447 s · raw 538.447 s

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: published Vals component protocol. Exact Vals task/version and source model. Requested effort, provider and diagnostics are retained with this observation; evaluation date and checkpoint hash are unpublished. Source update date is not execution date. Quality standard error is not a confidence interval. A source revision does not by itself establish a model update. Workload diagnostic; quality uncertainty does not describe its uncertainty. Not a token tariff or controlled efficiency ranking.

58.952% · raw 58.952 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: published Vals component protocol. Exact Vals task/version and source model. Requested effort, provider and diagnostics are retained with this observation; evaluation date and checkpoint hash are unpublished. Source update date is not execution date. Quality standard error is not a confidence interval. A source revision does not by itself establish a model update.

50.088% · raw 50.088 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: published Vals component protocol. Exact Vals task/version and source model. Requested effort, provider and diagnostics are retained with this observation; evaluation date and checkpoint hash are unpublished. Source update date is not execution date. Quality standard error is not a confidence interval. A source revision does not by itself establish a model update.

53.078% · raw 53.078 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: published Vals component protocol. Exact Vals task/version and source model. Requested effort, provider and diagnostics are retained with this observation; evaluation date and checkpoint hash are unpublished. Source update date is not execution date. Quality standard error is not a confidence interval. A source revision does not by itself establish a model update.

77.551% · raw 77.551 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: published Vals component protocol. Exact Vals task/version and source model. Requested effort, provider and diagnostics are retained with this observation; evaluation date and checkpoint hash are unpublished. Source update date is not execution date. Quality standard error is not a confidence interval. A source revision does not by itself establish a model update.

77.903% · raw 77.903 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: published Vals component protocol. Exact Vals task/version and source model. Requested effort, provider and diagnostics are retained with this observation; evaluation date and checkpoint hash are unpublished. Source update date is not execution date. Quality standard error is not a confidence interval. A source revision does not by itself establish a model update.

81.489% · raw 81.489 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: published Vals component protocol. Exact Vals task/version and source model. Requested effort, provider and diagnostics are retained with this observation; evaluation date and checkpoint hash are unpublished. Source update date is not execution date. Quality standard error is not a confidence interval. A source revision does not by itself establish a model update.

4.545% · raw 4.545 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: published Vals component protocol. Exact Vals task/version and source model. Requested effort, provider and diagnostics are retained with this observation; evaluation date and checkpoint hash are unpublished. Source update date is not execution date. Quality standard error is not a confidence interval. A source revision does not by itself establish a model update.

48.224% · raw 48.224 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: published Vals component protocol. Exact Vals task/version and source model. Requested effort, provider and diagnostics are retained with this observation; evaluation date and checkpoint hash are unpublished. Source update date is not execution date. Quality standard error is not a confidence interval. A source revision does not by itself establish a model update.

41.461% · raw 41.461 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: published Vals component protocol. Exact Vals task/version and source model. Requested effort, provider and diagnostics are retained with this observation; evaluation date and checkpoint hash are unpublished. Source update date is not execution date. Quality standard error is not a confidence interval. A source revision does not by itself establish a model update.

25.481% · raw 25.481 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: published Vals component protocol. Exact Vals task/version and source model. Requested effort, provider and diagnostics are retained with this observation; evaluation date and checkpoint hash are unpublished. Source update date is not execution date. Quality standard error is not a confidence interval. A source revision does not by itself establish a model update.

81.493% · raw 81.493 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: published Vals component protocol. Exact Vals task/version and source model. Requested effort, provider and diagnostics are retained with this observation; evaluation date and checkpoint hash are unpublished. Source update date is not execution date. Quality standard error is not a confidence interval. A source revision does not by itself establish a model update.

$0.141548/task · raw 0.141548 USD/task

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: published Vals component protocol. Exact Vals task/version and source model. Requested effort, provider and diagnostics are retained with this observation; evaluation date and checkpoint hash are unpublished. Source update date is not execution date. Quality standard error is not a confidence interval. A source revision does not by itself establish a model update. Workload diagnostic; quality uncertainty does not describe its uncertainty. Not a token tariff or controlled efficiency ranking.

774.823 s · raw 774.823 s

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: published Vals component protocol. Exact Vals task/version and source model. Requested effort, provider and diagnostics are retained with this observation; evaluation date and checkpoint hash are unpublished. Source update date is not execution date. Quality standard error is not a confidence interval. A source revision does not by itself establish a model update. Workload diagnostic; quality uncertainty does not describe its uncertainty. Not a token tariff or controlled efficiency ranking.

28.788% · raw 28.788 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: published Vals component protocol. Exact Vals task/version and source model. Requested effort, provider and diagnostics are retained with this observation; evaluation date and checkpoint hash are unpublished. Source update date is not execution date. Quality standard error is not a confidence interval. A source revision does not by itself establish a model update.

34.884% · raw 34.884 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: published Vals component protocol. Exact Vals task/version and source model. Requested effort, provider and diagnostics are retained with this observation; evaluation date and checkpoint hash are unpublished. Source update date is not execution date. Quality standard error is not a confidence interval. A source revision does not by itself establish a model update.

22.222% · raw 22.222 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: published Vals component protocol. Exact Vals task/version and source model. Requested effort, provider and diagnostics are retained with this observation; evaluation date and checkpoint hash are unpublished. Source update date is not execution date. Quality standard error is not a confidence interval. A source revision does not by itself establish a model update.

10.526% · raw 10.526 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: published Vals component protocol. Exact Vals task/version and source model. Requested effort, provider and diagnostics are retained with this observation; evaluation date and checkpoint hash are unpublished. Source update date is not execution date. Quality standard error is not a confidence interval. A source revision does not by itself establish a model update.

24.561% · raw 24.561 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: published Vals component protocol. Exact Vals task/version and source model. Requested effort, provider and diagnostics are retained with this observation; evaluation date and checkpoint hash are unpublished. Source update date is not execution date. Quality standard error is not a confidence interval. A source revision does not by itself establish a model update.

4.545% · raw 4.545 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: published Vals component protocol. Exact Vals task/version and source model. Requested effort, provider and diagnostics are retained with this observation; evaluation date and checkpoint hash are unpublished. Source update date is not execution date. Quality standard error is not a confidence interval. A source revision does not by itself establish a model update.

30.000% · raw 30 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: published Vals component protocol. Exact Vals task/version and source model. Requested effort, provider and diagnostics are retained with this observation; evaluation date and checkpoint hash are unpublished. Source update date is not execution date. Quality standard error is not a confidence interval. A source revision does not by itself establish a model update.

18.182% · raw 18.182 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: published Vals component protocol. Exact Vals task/version and source model. Requested effort, provider and diagnostics are retained with this observation; evaluation date and checkpoint hash are unpublished. Source update date is not execution date. Quality standard error is not a confidence interval. A source revision does not by itself establish a model update.

78.924% · raw 78.924 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: published Vals component protocol. Exact Vals task/version and source model. Requested effort, provider and diagnostics are retained with this observation; evaluation date and checkpoint hash are unpublished. Source update date is not execution date. Quality standard error is not a confidence interval. A source revision does not by itself establish a model update.

82.955% · raw 82.955 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: published Vals component protocol. Exact Vals task/version and source model. Requested effort, provider and diagnostics are retained with this observation; evaluation date and checkpoint hash are unpublished. Source update date is not execution date. Quality standard error is not a confidence interval. A source revision does not by itself establish a model update.

78.197% · raw 78.197 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: published Vals component protocol. Exact Vals task/version and source model. Requested effort, provider and diagnostics are retained with this observation; evaluation date and checkpoint hash are unpublished. Source update date is not execution date. Quality standard error is not a confidence interval. A source revision does not by itself establish a model update.

81.523% · raw 81.523 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: published Vals component protocol. Exact Vals task/version and source model. Requested effort, provider and diagnostics are retained with this observation; evaluation date and checkpoint hash are unpublished. Source update date is not execution date. Quality standard error is not a confidence interval. A source revision does not by itself establish a model update.

80.858% · raw 80.858 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: published Vals component protocol. Exact Vals task/version and source model. Requested effort, provider and diagnostics are retained with this observation; evaluation date and checkpoint hash are unpublished. Source update date is not execution date. Quality standard error is not a confidence interval. A source revision does not by itself establish a model update.

78.032% · raw 78.032 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: published Vals component protocol. Exact Vals task/version and source model. Requested effort, provider and diagnostics are retained with this observation; evaluation date and checkpoint hash are unpublished. Source update date is not execution date. Quality standard error is not a confidence interval. A source revision does not by itself establish a model update.

68.892% · raw 68.892 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: published Vals component protocol. Exact Vals task/version and source model. Requested effort, provider and diagnostics are retained with this observation; evaluation date and checkpoint hash are unpublished. Source update date is not execution date. Quality standard error is not a confidence interval. A source revision does not by itself establish a model update.

84.385% · raw 84.385 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: published Vals component protocol. Exact Vals task/version and source model. Requested effort, provider and diagnostics are retained with this observation; evaluation date and checkpoint hash are unpublished. Source update date is not execution date. Quality standard error is not a confidence interval. A source revision does not by itself establish a model update.

76.087% · raw 76.087 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: published Vals component protocol. Exact Vals task/version and source model. Requested effort, provider and diagnostics are retained with this observation; evaluation date and checkpoint hash are unpublished. Source update date is not execution date. Quality standard error is not a confidence interval. A source revision does not by itself establish a model update.

$1.355595/task · raw 1.355595 USD/task

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: published Vals component protocol. Exact Vals task/version and source model. Requested effort, provider and diagnostics are retained with this observation; evaluation date and checkpoint hash are unpublished. Source update date is not execution date. Quality standard error is not a confidence interval. A source revision does not by itself establish a model update. Workload diagnostic; quality uncertainty does not describe its uncertainty. Not a token tariff or controlled efficiency ranking.

1076.499 s · raw 1076.499 s

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: published Vals component protocol. Exact Vals task/version and source model. Requested effort, provider and diagnostics are retained with this observation; evaluation date and checkpoint hash are unpublished. Source update date is not execution date. Quality standard error is not a confidence interval. A source revision does not by itself establish a model update. Workload diagnostic; quality uncertainty does not describe its uncertainty. Not a token tariff or controlled efficiency ranking.

3.010% · raw 3.01 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: published Vals component protocol. Exact Vals task/version and source model. Requested effort, provider and diagnostics are retained with this observation; evaluation date and checkpoint hash are unpublished. Source update date is not execution date. Quality standard error is not a confidence interval. A source revision does not by itself establish a model update.

32.667% · raw 32.667 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: published Vals component protocol. Exact Vals task/version and source model. Requested effort, provider and diagnostics are retained with this observation; evaluation date and checkpoint hash are unpublished. Source update date is not execution date. Quality standard error is not a confidence interval. A source revision does not by itself establish a model update.

2.031 points/5 · raw 2.031 points/5

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: published Vals component protocol. Exact Vals task/version and source model. Requested effort, provider and diagnostics are retained with this observation; evaluation date and checkpoint hash are unpublished. Source update date is not execution date. Quality standard error is not a confidence interval. A source revision does not by itself establish a model update.

$0.091767/task · raw 0.091767 USD/task

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: published Vals component protocol. Exact Vals task/version and source model. Requested effort, provider and diagnostics are retained with this observation; evaluation date and checkpoint hash are unpublished. Source update date is not execution date. Quality standard error is not a confidence interval. A source revision does not by itself establish a model update. Workload diagnostic; quality uncertainty does not describe its uncertainty. Not a token tariff or controlled efficiency ranking.

385.252 s · raw 385.252 s

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: published Vals component protocol. Exact Vals task/version and source model. Requested effort, provider and diagnostics are retained with this observation; evaluation date and checkpoint hash are unpublished. Source update date is not execution date. Quality standard error is not a confidence interval. A source revision does not by itself establish a model update. Workload diagnostic; quality uncertainty does not describe its uncertainty. Not a token tariff or controlled efficiency ranking.

12.558% · raw 12.558 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: published Vals component protocol. Exact Vals task/version and source model. Requested effort, provider and diagnostics are retained with this observation; evaluation date and checkpoint hash are unpublished. Source update date is not execution date. Quality standard error is not a confidence interval. A source revision does not by itself establish a model update.

14.000% · raw 14 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: published Vals component protocol. Exact Vals task/version and source model. Requested effort, provider and diagnostics are retained with this observation; evaluation date and checkpoint hash are unpublished. Source update date is not execution date. Quality standard error is not a confidence interval. A source revision does not by itself establish a model update.

26.333% · raw 26.333 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: published Vals component protocol. Exact Vals task/version and source model. Requested effort, provider and diagnostics are retained with this observation; evaluation date and checkpoint hash are unpublished. Source update date is not execution date. Quality standard error is not a confidence interval. A source revision does not by itself establish a model update.

23.667% · raw 23.667 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: published Vals component protocol. Exact Vals task/version and source model. Requested effort, provider and diagnostics are retained with this observation; evaluation date and checkpoint hash are unpublished. Source update date is not execution date. Quality standard error is not a confidence interval. A source revision does not by itself establish a model update.

10.333% · raw 10.333 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: published Vals component protocol. Exact Vals task/version and source model. Requested effort, provider and diagnostics are retained with this observation; evaluation date and checkpoint hash are unpublished. Source update date is not execution date. Quality standard error is not a confidence interval. A source revision does not by itself establish a model update.

6.667% · raw 6.667 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: published Vals component protocol. Exact Vals task/version and source model. Requested effort, provider and diagnostics are retained with this observation; evaluation date and checkpoint hash are unpublished. Source update date is not execution date. Quality standard error is not a confidence interval. A source revision does not by itself establish a model update.

41.461% · raw 41.461 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: published Vals component protocol. Exact Vals task/version and source model. Requested effort, provider and diagnostics are retained with this observation; evaluation date and checkpoint hash are unpublished. Source update date is not execution date. Quality standard error is not a confidence interval. A source revision does not by itself establish a model update.

43.850% · raw 43.85 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: published Vals component protocol. Exact Vals task/version and source model. Requested effort, provider and diagnostics are retained with this observation; evaluation date and checkpoint hash are unpublished. Source update date is not execution date. Quality standard error is not a confidence interval. A source revision does not by itself establish a model update.

32.085% · raw 32.085 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: published Vals component protocol. Exact Vals task/version and source model. Requested effort, provider and diagnostics are retained with this observation; evaluation date and checkpoint hash are unpublished. Source update date is not execution date. Quality standard error is not a confidence interval. A source revision does not by itself establish a model update.

47.453% · raw 47.453 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: published Vals component protocol. Exact Vals task/version and source model. Requested effort, provider and diagnostics are retained with this observation; evaluation date and checkpoint hash are unpublished. Source update date is not execution date. Quality standard error is not a confidence interval. A source revision does not by itself establish a model update.

51.226% · raw 51.226 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: published Vals component protocol. Exact Vals task/version and source model. Requested effort, provider and diagnostics are retained with this observation; evaluation date and checkpoint hash are unpublished. Source update date is not execution date. Quality standard error is not a confidence interval. A source revision does not by itself establish a model update.

42.965% · raw 42.965 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: published Vals component protocol. Exact Vals task/version and source model. Requested effort, provider and diagnostics are retained with this observation; evaluation date and checkpoint hash are unpublished. Source update date is not execution date. Quality standard error is not a confidence interval. A source revision does not by itself establish a model update.

30.152% · raw 30.152 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: published Vals component protocol. Exact Vals task/version and source model. Requested effort, provider and diagnostics are retained with this observation; evaluation date and checkpoint hash are unpublished. Source update date is not execution date. Quality standard error is not a confidence interval. A source revision does not by itself establish a model update.

$0.321679/task · raw 0.321679 USD/task

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: published Vals component protocol. Exact Vals task/version and source model. Requested effort, provider and diagnostics are retained with this observation; evaluation date and checkpoint hash are unpublished. Source update date is not execution date. Quality standard error is not a confidence interval. A source revision does not by itself establish a model update. Workload diagnostic; quality uncertainty does not describe its uncertainty. Not a token tariff or controlled efficiency ranking.

705.115 s · raw 705.115 s

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: published Vals component protocol. Exact Vals task/version and source model. Requested effort, provider and diagnostics are retained with this observation; evaluation date and checkpoint hash are unpublished. Source update date is not execution date. Quality standard error is not a confidence interval. A source revision does not by itself establish a model update. Workload diagnostic; quality uncertainty does not describe its uncertainty. Not a token tariff or controlled efficiency ranking.

0.000% · raw 0 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: published Vals component protocol. Exact Vals task/version and source model. Requested effort, provider and diagnostics are retained with this observation; evaluation date and checkpoint hash are unpublished. Source update date is not execution date. Quality standard error is not a confidence interval. A source revision does not by itself establish a model update.

0.000% · raw 0 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: published Vals component protocol. Exact Vals task/version and source model. Requested effort, provider and diagnostics are retained with this observation; evaluation date and checkpoint hash are unpublished. Source update date is not execution date. Quality standard error is not a confidence interval. A source revision does not by itself establish a model update.

0.000% · raw 0 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: published Vals component protocol. Exact Vals task/version and source model. Requested effort, provider and diagnostics are retained with this observation; evaluation date and checkpoint hash are unpublished. Source update date is not execution date. Quality standard error is not a confidence interval. A source revision does not by itself establish a model update.

0.000% · raw 0 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: published Vals component protocol. Exact Vals task/version and source model. Requested effort, provider and diagnostics are retained with this observation; evaluation date and checkpoint hash are unpublished. Source update date is not execution date. Quality standard error is not a confidence interval. A source revision does not by itself establish a model update.

0.000% · raw 0 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: published Vals component protocol. Exact Vals task/version and source model. Requested effort, provider and diagnostics are retained with this observation; evaluation date and checkpoint hash are unpublished. Source update date is not execution date. Quality standard error is not a confidence interval. A source revision does not by itself establish a model update.

0.000% · raw 0 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: published Vals component protocol. Exact Vals task/version and source model. Requested effort, provider and diagnostics are retained with this observation; evaluation date and checkpoint hash are unpublished. Source update date is not execution date. Quality standard error is not a confidence interval. A source revision does not by itself establish a model update.

0.000% · raw 0 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: published Vals component protocol. Exact Vals task/version and source model. Requested effort, provider and diagnostics are retained with this observation; evaluation date and checkpoint hash are unpublished. Source update date is not execution date. Quality standard error is not a confidence interval. A source revision does not by itself establish a model update.

$0.432264/task · raw 0.432264 USD/task

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: OpenHands. Exact Vals task/version and source model. Requested effort, provider and diagnostics are retained with this observation; evaluation date and checkpoint hash are unpublished. Source update date is not execution date. Quality standard error is not a confidence interval. A source revision does not by itself establish a model update. Workload diagnostic; quality uncertainty does not describe its uncertainty. Not a token tariff or controlled efficiency ranking.

2949.469 s · raw 2949.469 s

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: OpenHands. Exact Vals task/version and source model. Requested effort, provider and diagnostics are retained with this observation; evaluation date and checkpoint hash are unpublished. Source update date is not execution date. Quality standard error is not a confidence interval. A source revision does not by itself establish a model update. Workload diagnostic; quality uncertainty does not describe its uncertainty. Not a token tariff or controlled efficiency ranking.

30 correct · raw 30 correct

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-07

Harness: Plotly Studio Embedded, one run on 43 private English data-analytics questions in a synthetic wind-energy warehouse. Claude Sonnet 5 judges against Plotly answer key. One or two questions can be noise; no error bars. Exact checkpoint hash, run date and effort unpublished. Company-operated analytics benchmark, distinct from provider results. Table row as displayed: “Inkling-Small† Jul 30, 2026 30 / 43 $1.93 16.4M 25 s / 32 s 23 min”. † marks open-weight models. Cost is at OpenRouter prices on the run day; the date column is the release date Plotly lists. Serving precision unstated.