Examenos

56.4% · raw 56.4 %

Headline · unknown effort · Own vendor

Source/record date: 2026-06-04

NVIDIA Nemotron 3 Ultra BF16 card (revision 77df655d5e9f), benchmark table, row “Terminal Bench 2.1”. Collected with the NeMo Evaluator SDK (NeMo Gym, NeMo Skills and Harbor harnesses); pinned settings in NVIDIA’s reproducibility tutorial (https://github.com/NVIDIA-NeMo/Evaluator/blob/main/examples/nemotron/nemotron-3-ultra/reproducibility.md). The card does not name the agent for this value; report Figure 17 gives per-agent runs (Terminus 2.0.0: 55.1). The technical report repeats this table as Table 10. Effort levels unpublished; execution date unpublished.

55.1% · raw 55.1 %

Alternative · unknown effort · Own vendor

Source/record date: 2026-06-04

NVIDIA technical report Figure 17 (page 65), agent × model matrix for Terminal-Bench 2.1, row “ultra”, agent column “Terminus (2.0.0)”: printed cell label read from the figure (crop in research-raw). The matrix’s average column (SWE-bench Verified 60.6, Terminal-Bench 2.1 48.9) is not stored. Different agent setup from the card’s headline value, so non-headline. Effort unpublished. Non-headline: the first-party row (56.4, https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16) holds this cell.

46.7% · raw 46.7 %

Alternative · unknown effort · Own vendor

Source/record date: 2026-06-04

NVIDIA technical report Figure 17 (page 65), agent × model matrix for Terminal-Bench 2.1, row “ultra”, agent column “OpenCode (1.14.33)”: printed cell label read from the figure (crop in research-raw). The matrix’s average column (SWE-bench Verified 60.6, Terminal-Bench 2.1 48.9) is not stored. Different agent setup from the card’s headline value, so non-headline. Effort unpublished. Non-headline: the first-party row (56.4, https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16) holds this cell.

52.1% · raw 52.1 %

Alternative · unknown effort · Own vendor

Source/record date: 2026-06-04

NVIDIA technical report Figure 17 (page 65), agent × model matrix for Terminal-Bench 2.1, row “ultra”, agent column “Pi”: printed cell label read from the figure (crop in research-raw). The matrix’s average column (SWE-bench Verified 60.6, Terminal-Bench 2.1 48.9) is not stored. Different agent setup from the card’s headline value, so non-headline. Effort unpublished. Non-headline: the first-party row (56.4, https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16) holds this cell.

47.2% · raw 47.2 %

Alternative · unknown effort · Own vendor

Source/record date: 2026-06-04

NVIDIA technical report Figure 17 (page 65), agent × model matrix for Terminal-Bench 2.1, row “ultra”, agent column “Claude (2.1.126)”: printed cell label read from the figure (crop in research-raw). The matrix’s average column (SWE-bench Verified 60.6, Terminal-Bench 2.1 48.9) is not stored. Different agent setup from the card’s headline value, so non-headline. Effort unpublished. Non-headline: the first-party row (56.4, https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16) holds this cell.

52.8% · raw 52.8 %

Alternative · unknown effort · Own vendor

Source/record date: 2026-06-04

NVIDIA technical report Figure 17 (page 65), agent × model matrix for Terminal-Bench 2.1, row “ultra”, agent column “Hermes”: printed cell label read from the figure (crop in research-raw). The matrix’s average column (SWE-bench Verified 60.6, Terminal-Bench 2.1 48.9) is not stored. Different agent setup from the card’s headline value, so non-headline. Effort unpublished. Non-headline: the first-party row (56.4, https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16) holds this cell.

52.6% · raw 52.6 %

Alternative · unknown effort · Own vendor

Source/record date: 2026-06-04

NVIDIA technical report Figure 17 (page 65), agent × model matrix for Terminal-Bench 2.1, row “ultra”, agent column “OpenHands (1.17.0)”: printed cell label read from the figure (crop in research-raw). The matrix’s average column (SWE-bench Verified 60.6, Terminal-Bench 2.1 48.9) is not stored. Different agent setup from the card’s headline value, so non-headline. Effort unpublished. Non-headline: the first-party row (56.4, https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16) holds this cell.

35.5% · raw 35.5 %

Alternative · unknown effort · Own vendor

Source/record date: 2026-06-04

NVIDIA technical report Figure 17 (page 65), agent × model matrix for Terminal-Bench 2.1, row “ultra”, agent column “Codex (0.128.0)”: printed cell label read from the figure (crop in research-raw). The matrix’s average column (SWE-bench Verified 60.6, Terminal-Bench 2.1 48.9) is not stored. Different agent setup from the card’s headline value, so non-headline. Effort unpublished. Non-headline: the first-party row (56.4, https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16) holds this cell.

54.8% · raw 54.78 %

Alternative · unknown effort · Own vendor

Source/record date: 2026-10-15

NVIDIA’s own Nemotron 3 Ultra result in its later Nemotron 3.5 Super and Lightning technical report (15 October 2026), Table 10, column “N3-Ultra 550B-A55B”, row “Terminal-Bench2.1”; the report does not state the precision. NeMo Gym; report Table 10 and §§3.8–3.9. Terminus 2 with a 3-hour task timeout. Execution date and effort unpublished. Non-headline: the first-party row (56.4, https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16) holds this cell.

54.8% · raw 54.8 %

Alternative · unknown effort · Own vendor

Source/record date: 2026-10-15

NVIDIA’s own Nemotron 3 Ultra result in the Nemotron 3.5 Super VL BF16 card (15 October 2026), column “Nemotron 3 Ultra”; the card does not state the precision. Exact released BF16 card table. Conflicts/rounding versus detailed report remain non-headline alternatives; detailed report supplies the precise protocol. Raw card label: Terminal-Bench 2.1. Execution date and effort unpublished. Non-headline: the first-party row (56.4, https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16) holds this cell.

46.7 score · raw 46.7 score

Headline · unknown effort · Own vendor

Source/record date: 2026-06-04

NVIDIA Nemotron 3 Ultra BF16 card (revision 77df655d5e9f), benchmark table, row “GDPVal”. Collected with the NeMo Evaluator SDK (NeMo Gym, NeMo Skills and Harbor harnesses); pinned settings in NVIDIA’s reproducibility tutorial (https://github.com/NVIDIA-NeMo/Evaluator/blob/main/examples/nemotron/nemotron-3-ultra/reproducibility.md). Stirrup agent in NeMo Gym, pairwise or rubric judging (Gemini 3.1 Pro judge in the reproduction config); a 0–100 score, not an Elo. The technical report repeats this table as Table 10. Effort levels unpublished; execution date unpublished.

70.7% · raw 70.7 %

Headline · unknown effort · Own vendor

Source/record date: 2026-06-04

NVIDIA Nemotron 3 Ultra BF16 card (revision 77df655d5e9f), benchmark table, row “SWE-Bench Verified”. Collected with the NeMo Evaluator SDK (NeMo Gym, NeMo Skills and Harbor harnesses); pinned settings in NVIDIA’s reproducibility tutorial (https://github.com/NVIDIA-NeMo/Evaluator/blob/main/examples/nemotron/nemotron-3-ultra/reproducibility.md). NeMo Evaluator v0.3 native SWE-bench run (reproducibility notes); the card does not name the agent. Report Figure 17 per-agent runs: OpenHands 70.3, OpenCode 67.3. The technical report repeats this table as Table 10. Effort levels unpublished; execution date unpublished.

65% · raw 65 %

Alternative · unknown effort · Own vendor

Source/record date: 2026-06-04

NVIDIA technical report Figure 17 (page 65), agent × model matrix for SWE-bench Verified, row “ultra”, agent column “Mini SWE Agent (2.3.0)”: printed cell label read from the figure (crop in research-raw). The matrix’s average column (SWE-bench Verified 60.6, Terminal-Bench 2.1 48.9) is not stored. Different agent from the card’s headline run, so non-headline. Effort unpublished. Non-headline: the first-party row (70.7, https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16) holds this cell.

70.4% · raw 70.4 %

Alternative · unknown effort · Own vendor

Source/record date: 2026-06-04

NVIDIA technical report Figure 17 (page 65), agent × model matrix for SWE-bench Verified, row “ultra”, agent column “Pi (v0.72.1)”: printed cell label read from the figure (crop in research-raw). The matrix’s average column (SWE-bench Verified 60.6, Terminal-Bench 2.1 48.9) is not stored. Different agent from the card’s headline run, so non-headline. Effort unpublished. Non-headline: the first-party row (70.7, https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16) holds this cell.

60.3% · raw 60.3 %

Alternative · unknown effort · Own vendor

Source/record date: 2026-06-04

NVIDIA technical report Figure 17 (page 65), agent × model matrix for SWE-bench Verified, row “ultra”, agent column “Claude (2.1.126)”: printed cell label read from the figure (crop in research-raw). The matrix’s average column (SWE-bench Verified 60.6, Terminal-Bench 2.1 48.9) is not stored. Different agent from the card’s headline run, so non-headline. Effort unpublished. Non-headline: the first-party row (70.7, https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16) holds this cell.

69.9% · raw 69.9 %

Alternative · unknown effort · Own vendor

Source/record date: 2026-06-04

NVIDIA technical report Figure 17 (page 65), agent × model matrix for SWE-bench Verified, row “ultra”, agent column “Hermes (v2026.4.30)”: printed cell label read from the figure (crop in research-raw). The matrix’s average column (SWE-bench Verified 60.6, Terminal-Bench 2.1 48.9) is not stored. Different agent from the card’s headline run, so non-headline. Effort unpublished. Non-headline: the first-party row (70.7, https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16) holds this cell.

21.1% · raw 21.1 %

Alternative · unknown effort · Own vendor

Source/record date: 2026-06-04

NVIDIA technical report Figure 17 (page 65), agent × model matrix for SWE-bench Verified, row “ultra”, agent column “Codex (0.128/0.135)”: printed cell label read from the figure (crop in research-raw). The matrix’s average column (SWE-bench Verified 60.6, Terminal-Bench 2.1 48.9) is not stored. Different agent from the card’s headline run, so non-headline. Effort unpublished. Non-headline: the first-party row (70.7, https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16) holds this cell.

67.7% · raw 67.7 %

Headline · unknown effort · Own vendor

Source/record date: 2026-06-04

NVIDIA Nemotron 3 Ultra BF16 card (revision 77df655d5e9f), benchmark table, row “SWE-Bench Multilingual”. Collected with the NeMo Evaluator SDK (NeMo Gym, NeMo Skills and Harbor harnesses); pinned settings in NVIDIA’s reproducibility tutorial (https://github.com/NVIDIA-NeMo/Evaluator/blob/main/examples/nemotron/nemotron-3-ultra/reproducibility.md). Agent not named on the card. The technical report repeats this table as Table 10. Effort levels unpublished; execution date unpublished.

56% · raw 56 %

Headline · unknown effort · Own vendor

Source/record date: 2026-06-04

NVIDIA Nemotron 3 Ultra BF16 card (revision 77df655d5e9f), benchmark table, row “ProfBench (Search)”. Collected with the NeMo Evaluator SDK (NeMo Gym, NeMo Skills and Harbor harnesses); pinned settings in NVIDIA’s reproducibility tutorial (https://github.com/NVIDIA-NeMo/Evaluator/blob/main/examples/nemotron/nemotron-3-ultra/reproducibility.md). NVIDIA lists this benchmark among those run with the official implementation or an internal scaffold rather than its open-source NeMo tools. The technical report repeats this table as Table 10. Effort levels unpublished; execution date unpublished.

90% · raw 90 %

Headline · unknown effort · Own vendor

Source/record date: 2026-06-04

NVIDIA Nemotron 3 Ultra BF16 card (revision 77df655d5e9f), benchmark table, row “PinchBench”. Collected with the NeMo Evaluator SDK (NeMo Gym, NeMo Skills and Harbor harnesses); pinned settings in NVIDIA’s reproducibility tutorial (https://github.com/NVIDIA-NeMo/Evaluator/blob/main/examples/nemotron/nemotron-3-ultra/reproducibility.md). NVIDIA lists this benchmark among those run with the official implementation or an internal scaffold rather than its open-source NeMo tools. The technical report repeats this table as Table 10. Effort levels unpublished; execution date unpublished.

81.5% · raw 81.5 %

Headline · unknown effort · Own vendor

Source/record date: 2026-06-04

NVIDIA Nemotron 3 Ultra BF16 card (revision 77df655d5e9f), benchmark table, row “TauBench V3 / Airline”. Collected with the NeMo Evaluator SDK (NeMo Gym, NeMo Skills and Harbor harnesses); pinned settings in NVIDIA’s reproducibility tutorial (https://github.com/NVIDIA-NeMo/Evaluator/blob/main/examples/nemotron/nemotron-3-ultra/reproducibility.md). NVIDIA lists this benchmark among those run with the official implementation or an internal scaffold rather than its open-source NeMo tools. The technical report repeats this table as Table 10. Effort levels unpublished; execution date unpublished.

86.4% · raw 86.4 %

Headline · unknown effort · Own vendor

Source/record date: 2026-06-04

NVIDIA Nemotron 3 Ultra BF16 card (revision 77df655d5e9f), benchmark table, row “TauBench V3 / Retail”. Collected with the NeMo Evaluator SDK (NeMo Gym, NeMo Skills and Harbor harnesses); pinned settings in NVIDIA’s reproducibility tutorial (https://github.com/NVIDIA-NeMo/Evaluator/blob/main/examples/nemotron/nemotron-3-ultra/reproducibility.md). NVIDIA lists this benchmark among those run with the official implementation or an internal scaffold rather than its open-source NeMo tools. The technical report repeats this table as Table 10. Effort levels unpublished; execution date unpublished.

92.9% · raw 92.9 %

Headline · unknown effort · Own vendor

Source/record date: 2026-06-04

NVIDIA Nemotron 3 Ultra BF16 card (revision 77df655d5e9f), benchmark table, row “TauBench V3 / Telecom”. Collected with the NeMo Evaluator SDK (NeMo Gym, NeMo Skills and Harbor harnesses); pinned settings in NVIDIA’s reproducibility tutorial (https://github.com/NVIDIA-NeMo/Evaluator/blob/main/examples/nemotron/nemotron-3-ultra/reproducibility.md). NVIDIA lists this benchmark among those run with the official implementation or an internal scaffold rather than its open-source NeMo tools. The technical report repeats this table as Table 10. Effort levels unpublished; execution date unpublished.

22.6% · raw 22.6 %

Headline · unknown effort · Own vendor

Source/record date: 2026-06-04

NVIDIA Nemotron 3 Ultra BF16 card (revision 77df655d5e9f), benchmark table, row “TauBench V3 / Banking”. Collected with the NeMo Evaluator SDK (NeMo Gym, NeMo Skills and Harbor harnesses); pinned settings in NVIDIA’s reproducibility tutorial (https://github.com/NVIDIA-NeMo/Evaluator/blob/main/examples/nemotron/nemotron-3-ultra/reproducibility.md). NVIDIA lists this benchmark among those run with the official implementation or an internal scaffold rather than its open-source NeMo tools. The technical report repeats this table as Table 10. Effort levels unpublished; execution date unpublished.

18.4% · raw 18.35 %

Alternative · unknown effort · Own vendor

Source/record date: 2026-10-15

NVIDIA’s own Nemotron 3 Ultra result in its later Nemotron 3.5 Super and Lightning technical report (15 October 2026), Table 10, column “N3-Ultra 550B-A55B”, row “Tau3Banking”; the report does not state the precision. NeMo Gym; report Table 10 and §§3.8–3.9. Execution date and effort unpublished. Non-headline: the first-party row (22.6, https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16) holds this cell.

70.9% · raw 70.9 %

Headline · unknown effort · Own vendor

Source/record date: 2026-06-04

NVIDIA Nemotron 3 Ultra BF16 card (revision 77df655d5e9f), benchmark table, row “TauBench V3 / Average”. Collected with the NeMo Evaluator SDK (NeMo Gym, NeMo Skills and Harbor harnesses); pinned settings in NVIDIA’s reproducibility tutorial (https://github.com/NVIDIA-NeMo/Evaluator/blob/main/examples/nemotron/nemotron-3-ultra/reproducibility.md). Mean of the four domains as printed. NVIDIA lists this benchmark among those run with the official implementation or an internal scaffold rather than its open-source NeMo tools. The technical report repeats this table as Table 10. Effort levels unpublished; execution date unpublished.

44.4% · raw 44.4 %

Headline · unknown effort · Own vendor

Source/record date: 2026-06-04

NVIDIA Nemotron 3 Ultra BF16 card (revision 77df655d5e9f), benchmark table, row “BrowseComp”. Collected with the NeMo Evaluator SDK (NeMo Gym, NeMo Skills and Harbor harnesses); pinned settings in NVIDIA’s reproducibility tutorial (https://github.com/NVIDIA-NeMo/Evaluator/blob/main/examples/nemotron/nemotron-3-ultra/reproducibility.md). BrowseComp with search. NVIDIA lists this benchmark among those run with the official implementation or an internal scaffold rather than its open-source NeMo tools. The technical report repeats this table as Table 10. Effort levels unpublished; execution date unpublished.

63% · raw 63 %

Alternative · unknown effort · Peer vendor

Source/record date: 2026-07-30

As reported by Thinking Machines in its Inkling-Small post (30 July 2026), column “Nemotron 3 Ultra”, row “BrowseComp w/ ctx management”. NVIDIA publishes no value for this exact setup, so this is Thinking Machines’ own run of the peer; precision, reasoning mode and effort for the peer are unpublished. With context management; harness unpublished. Not NVIDIA’s result and not an independent evaluator record. Non-headline: the first-party row (44.4, https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16) holds this cell.

60.1% · raw 60.1 %

Headline · unknown effort · Own vendor

Source/record date: 2026-06-04

NVIDIA Nemotron 3 Ultra BF16 card (revision 77df655d5e9f), benchmark table, row “Vals.ai Financial Agent 1.1 / without web search”. Collected with the NeMo Evaluator SDK (NeMo Gym, NeMo Skills and Harbor harnesses); pinned settings in NVIDIA’s reproducibility tutorial (https://github.com/NVIDIA-NeMo/Evaluator/blob/main/examples/nemotron/nemotron-3-ultra/reproducibility.md). NVIDIA lists this benchmark among those run with the official implementation or an internal scaffold rather than its open-source NeMo tools. NVIDIA’s run, not a Vals-run result. The technical report repeats this table as Table 10. Effort levels unpublished; execution date unpublished.

53.7% · raw 53.7 %

Headline · unknown effort · Own vendor

Source/record date: 2026-06-04

NVIDIA Nemotron 3 Ultra BF16 card (revision 77df655d5e9f), benchmark table, row “Vals.ai Financial Agent 1.1 / with web search”. Collected with the NeMo Evaluator SDK (NeMo Gym, NeMo Skills and Harbor harnesses); pinned settings in NVIDIA’s reproducibility tutorial (https://github.com/NVIDIA-NeMo/Evaluator/blob/main/examples/nemotron/nemotron-3-ultra/reproducibility.md). NVIDIA lists this benchmark among those run with the official implementation or an internal scaffold rather than its open-source NeMo tools. NVIDIA’s run, not a Vals-run result. The technical report repeats this table as Table 10. Effort levels unpublished; execution date unpublished.

570 points · raw 570 points

Headline · unknown effort · Own vendor

Source/record date: 2026-06-04

NVIDIA Nemotron 3 Ultra BF16 card (revision 77df655d5e9f), benchmark table, row “IOI 2025”. Collected with the NeMo Evaluator SDK (NeMo Gym, NeMo Skills and Harbor harnesses); pinned settings in NVIDIA’s reproducibility tutorial (https://github.com/NVIDIA-NeMo/Evaluator/blob/main/examples/nemotron/nemotron-3-ultra/reproducibility.md). Official contest points out of 600; the report places it between the second- and third-ranked human contestants on the IOI 2025 scoreboard. The technical report repeats this table as Table 10. Effort levels unpublished; execution date unpublished.

89% · raw 89 %

Headline · unknown effort · Own vendor

Source/record date: 2026-06-04

NVIDIA Nemotron 3 Ultra BF16 card (revision 77df655d5e9f), benchmark table, row “LiveCodeBench (v6)”. Collected with the NeMo Evaluator SDK (NeMo Gym, NeMo Skills and Harbor harnesses); pinned settings in NVIDIA’s reproducibility tutorial (https://github.com/NVIDIA-NeMo/Evaluator/blob/main/examples/nemotron/nemotron-3-ultra/reproducibility.md). The technical report repeats this table as Table 10. Effort levels unpublished; execution date unpublished.

89.2% · raw 89.21 %

Alternative · unknown effort · Own vendor

Source/record date: 2026-10-15

NVIDIA’s own Nemotron 3 Ultra result in its later Nemotron 3.5 Super and Lightning technical report (15 October 2026), Table 10, column “N3-Ultra 550B-A55B”, row “LiveCodeBenchv6”; the report does not state the precision. NeMo Gym; report Table 10 and §§3.8–3.9. Execution date and effort unpublished. Non-headline: the first-party row (89.0, https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16) holds this cell.

88.6% · raw 88.6 %

Headline · unknown effort · Own vendor

Source/record date: 2026-06-04

NVIDIA Nemotron 3 Ultra BF16 card (revision 77df655d5e9f), benchmark table, row “IMOAnswerBench (no tools)”. Collected with the NeMo Evaluator SDK (NeMo Gym, NeMo Skills and Harbor harnesses); pinned settings in NVIDIA’s reproducibility tutorial (https://github.com/NVIDIA-NeMo/Evaluator/blob/main/examples/nemotron/nemotron-3-ultra/reproducibility.md). The technical report repeats this table as Table 10. Effort levels unpublished; execution date unpublished.

92.3% · raw 92.3 %

Headline · unknown effort · Own vendor

Source/record date: 2026-06-04

NVIDIA Nemotron 3 Ultra BF16 card (revision 77df655d5e9f), benchmark table, row “IMOAnswerBench (with tools)”. Collected with the NeMo Evaluator SDK (NeMo Gym, NeMo Skills and Harbor harnesses); pinned settings in NVIDIA’s reproducibility tutorial (https://github.com/NVIDIA-NeMo/Evaluator/blob/main/examples/nemotron/nemotron-3-ultra/reproducibility.md). Python tool for computation and verification. The technical report repeats this table as Table 10. Effort levels unpublished; execution date unpublished.

74.9% · raw 74.9 %

Headline · unknown effort · Own vendor

Source/record date: 2026-06-04

NVIDIA Nemotron 3 Ultra BF16 card (revision 77df655d5e9f), benchmark table, row “Apex-Shortlist (no tools)”. Collected with the NeMo Evaluator SDK (NeMo Gym, NeMo Skills and Harbor harnesses); pinned settings in NVIDIA’s reproducibility tutorial (https://github.com/NVIDIA-NeMo/Evaluator/blob/main/examples/nemotron/nemotron-3-ultra/reproducibility.md). The technical report repeats this table as Table 10. Effort levels unpublished; execution date unpublished.

75% · raw 75 %

Alternative · unknown effort · Own vendor

Source/record date: 2026-10-15

NVIDIA’s own Nemotron 3 Ultra result in its later Nemotron 3.5 Super and Lightning technical report (15 October 2026), Table 10, column “N3-Ultra 550B-A55B”, row “Apex-Shortlist”; the report does not state the precision. NeMo Gym; report Table 10 and §§3.8–3.9. APEX Shortlist 16 repeats, no tools. Execution date and effort unpublished. Non-headline: the first-party row (74.9, https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16) holds this cell.

84.8% · raw 84.8 %

Headline · unknown effort · Own vendor

Source/record date: 2026-06-04

NVIDIA Nemotron 3 Ultra BF16 card (revision 77df655d5e9f), benchmark table, row “Apex-Shortlist (with tools)”. Collected with the NeMo Evaluator SDK (NeMo Gym, NeMo Skills and Harbor harnesses); pinned settings in NVIDIA’s reproducibility tutorial (https://github.com/NVIDIA-NeMo/Evaluator/blob/main/examples/nemotron/nemotron-3-ultra/reproducibility.md). Python tool for computation and verification. The technical report repeats this table as Table 10. Effort levels unpublished; execution date unpublished.

87% · raw 87 %

Headline · unknown effort · Own vendor

Source/record date: 2026-06-04

NVIDIA Nemotron 3 Ultra BF16 card (revision 77df655d5e9f), benchmark table, row “GPQA (no tools)”. Collected with the NeMo Evaluator SDK (NeMo Gym, NeMo Skills and Harbor harnesses); pinned settings in NVIDIA’s reproducibility tutorial (https://github.com/NVIDIA-NeMo/Evaluator/blob/main/examples/nemotron/nemotron-3-ultra/reproducibility.md). Card label “GPQA (no tools)”; NVIDIA’s reproducibility notes run GPQA Diamond. The technical report repeats this table as Table 10. Effort levels unpublished; execution date unpublished.

86.7% · raw 86.67 %

Alternative · unknown effort · Own vendor

Source/record date: 2026-06-04

NVIDIA technical report Table 17 (page 47), BF16 reference column, row “GPQA”: pass@1 86.67, mean completion length 14,408 tokens. A separate run from the card table, made to compare the NVFP4 checkpoint; the W4A16 and NVFP4 columns describe that other checkpoint and are not stored. The card’s GPQA row is GPQA Diamond per NVIDIA’s reproducibility notes. Effort unpublished. Non-headline: the first-party row (87.0, https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16) holds this cell.

87% · raw 86.99 %

Alternative · unknown effort · Own vendor

Source/record date: 2026-10-15

NVIDIA’s own Nemotron 3 Ultra result in its later Nemotron 3.5 Super and Lightning technical report (15 October 2026), Table 10, column “N3-Ultra 550B-A55B”, row “GPQAD”; the report does not state the precision. NeMo Gym; report Table 10 and §§3.8–3.9. GPQA averages 8 repeats. Execution date and effort unpublished. Non-headline: the first-party row (87.0, https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16) holds this cell.

44.6% · raw 44.6 %

Headline · unknown effort · Own vendor

Source/record date: 2026-06-04

NVIDIA Nemotron 3 Ultra BF16 card (revision 77df655d5e9f), benchmark table, row “SciCode (subtask)”. Collected with the NeMo Evaluator SDK (NeMo Gym, NeMo Skills and Harbor harnesses); pinned settings in NVIDIA’s reproducibility tutorial (https://github.com/NVIDIA-NeMo/Evaluator/blob/main/examples/nemotron/nemotron-3-ultra/reproducibility.md). The technical report repeats this table as Table 10. Effort levels unpublished; execution date unpublished.

42% · raw 41.96 %

Alternative · unknown effort · Own vendor

Source/record date: 2026-10-15

NVIDIA’s own Nemotron 3 Ultra result in its later Nemotron 3.5 Super and Lightning technical report (15 October 2026), Table 10, column “N3-Ultra 550B-A55B”, row “SciCode(subtask)”; the report does not state the precision. NeMo Gym; report Table 10 and §§3.8–3.9. SciCode uses scientist background / 8 repeats; subtask accuracy. Execution date and effort unpublished. Non-headline: the first-party row (44.6, https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16) holds this cell.

26.7% · raw 26.7 %

Headline · unknown effort · Own vendor

Source/record date: 2026-06-04

NVIDIA Nemotron 3 Ultra BF16 card (revision 77df655d5e9f), benchmark table, row “HLE (no tools)”. Collected with the NeMo Evaluator SDK (NeMo Gym, NeMo Skills and Harbor harnesses); pinned settings in NVIDIA’s reproducibility tutorial (https://github.com/NVIDIA-NeMo/Evaluator/blob/main/examples/nemotron/nemotron-3-ultra/reproducibility.md). NVIDIA’s reproduction config runs HLE as NeMo Skills task ns_hle_aa (the Artificial Analysis text-only variant) with a strict judge. The technical report repeats this table as Table 10. Effort levels unpublished; execution date unpublished.

26.9% · raw 26.92 %

Alternative · unknown effort · Own vendor

Source/record date: 2026-06-04

NVIDIA technical report Table 17 (page 47), BF16 reference column, row “HLE”: pass@1 26.92, mean completion length 37,479 tokens. A separate run from the card table, made to compare the NVFP4 checkpoint; the W4A16 and NVFP4 columns describe that other checkpoint and are not stored. Table label “HLE”; NVIDIA’s reproduction runs the AA text-only variant. Effort unpublished. Non-headline: the first-party row (26.7, https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16) holds this cell.

29.2% · raw 29.24 %

Alternative · unknown effort · Own vendor

Source/record date: 2026-10-15

NVIDIA’s own Nemotron 3 Ultra result in its later Nemotron 3.5 Super and Lightning technical report (15 October 2026), Table 10, column “N3-Ultra 550B-A55B”, row “HLE(notools,text)”; the report does not state the precision. NeMo Gym; report Table 10 and §§3.8–3.9. HLE text-only has 2,158 questions. Execution date and effort unpublished. Non-headline: the first-party row (26.7, https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16) holds this cell.

37.4% · raw 37.4 %

Headline · unknown effort · Own vendor

Source/record date: 2026-06-04

NVIDIA Nemotron 3 Ultra BF16 card (revision 77df655d5e9f), benchmark table, row “HLE (with tools)”. Collected with the NeMo Evaluator SDK (NeMo Gym, NeMo Skills and Harbor harnesses); pinned settings in NVIDIA’s reproducibility tutorial (https://github.com/NVIDIA-NeMo/Evaluator/blob/main/examples/nemotron/nemotron-3-ultra/reproducibility.md). NVIDIA’s later Nemotron 3.5 report labels this Ultra result “HLE (with tools, text)” at 37.40. The technical report repeats this table as Table 10. Effort levels unpublished; execution date unpublished.

37.4% · raw 37.4 %

Alternative · unknown effort · Own vendor

Source/record date: 2026-10-15

NVIDIA’s own Nemotron 3 Ultra result in its later Nemotron 3.5 Super and Lightning technical report (15 October 2026), Table 10, column “N3-Ultra 550B-A55B”, row “HLE(withtools,text)”; the report does not state the precision. NeMo Gym; report Table 10 and §§3.8–3.9. HLE text-only has 2,158 questions. Execution date and effort unpublished. Non-headline: the first-party row (37.4, https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16) holds this cell.

3.1% · raw 3.1 %

Headline · unknown effort · Own vendor

Source/record date: 2026-06-04

NVIDIA Nemotron 3 Ultra BF16 card (revision 77df655d5e9f), benchmark table, row “CritPt (no tools)”. Collected with the NeMo Evaluator SDK (NeMo Gym, NeMo Skills and Harbor harnesses); pinned settings in NVIDIA’s reproducibility tutorial (https://github.com/NVIDIA-NeMo/Evaluator/blob/main/examples/nemotron/nemotron-3-ultra/reproducibility.md). The technical report repeats this table as Table 10. Effort levels unpublished; execution date unpublished.

3.1% · raw 3.14 %

Alternative · unknown effort · Own vendor

Source/record date: 2026-10-15

NVIDIA’s own Nemotron 3 Ultra result in its later Nemotron 3.5 Super and Lightning technical report (15 October 2026), Table 10, column “N3-Ultra 550B-A55B”, row “CritPt”; the report does not state the precision. NeMo Gym; report Table 10 and §§3.8–3.9. Execution date and effort unpublished. Non-headline: the first-party row (3.1, https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16) holds this cell.

86.8% · raw 86.8 %

Headline · unknown effort · Own vendor

Source/record date: 2026-06-04

NVIDIA Nemotron 3 Ultra BF16 card (revision 77df655d5e9f), benchmark table, row “MMLU-Pro”. Collected with the NeMo Evaluator SDK (NeMo Gym, NeMo Skills and Harbor harnesses); pinned settings in NVIDIA’s reproducibility tutorial (https://github.com/NVIDIA-NeMo/Evaluator/blob/main/examples/nemotron/nemotron-3-ultra/reproducibility.md). The technical report repeats this table as Table 10. Effort levels unpublished; execution date unpublished.

24.1% · raw 24.1 %

Headline · unknown effort · Own vendor

Source/record date: 2026-06-04

NVIDIA Nemotron 3 Ultra BF16 card (revision 77df655d5e9f), benchmark table, row “OmniScience Accuracy”. Collected with the NeMo Evaluator SDK (NeMo Gym, NeMo Skills and Harbor harnesses); pinned settings in NVIDIA’s reproducibility tutorial (https://github.com/NVIDIA-NeMo/Evaluator/blob/main/examples/nemotron/nemotron-3-ultra/reproducibility.md). AA-Omniscience run by NVIDIA with a Gemini judge (reproducibility notes), not Artificial Analysis’s own evaluation; question set not stated on the card. The technical report repeats this table as Table 10. Effort levels unpublished; execution date unpublished.

24.4% · raw 24.38 %

Alternative · unknown effort · Own vendor

Source/record date: 2026-06-04

NVIDIA technical report Table 17 (page 47), BF16 reference column, row “AA-Omniscience”: pass@1 24.38, mean completion length 1,140 tokens. A separate run from the card table, made to compare the NVFP4 checkpoint; the W4A16 and NVFP4 columns describe that other checkpoint and are not stored. Table label “AA-Omniscience”; the value tracks the card’s accuracy row (24.1), not its non-hallucination row (78.7), so it is stored as accuracy. Effort unpublished. Non-headline: the first-party row (24.1, https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16) holds this cell.

78.7% · raw 78.7 %

Headline · unknown effort · Own vendor

Source/record date: 2026-06-04

NVIDIA Nemotron 3 Ultra BF16 card (revision 77df655d5e9f), benchmark table, row “OmniScience Non-Hallucination”. Collected with the NeMo Evaluator SDK (NeMo Gym, NeMo Skills and Harbor harnesses); pinned settings in NVIDIA’s reproducibility tutorial (https://github.com/NVIDIA-NeMo/Evaluator/blob/main/examples/nemotron/nemotron-3-ultra/reproducibility.md). AA-Omniscience run by NVIDIA with a Gemini judge (reproducibility notes), not Artificial Analysis’s own evaluation; question set not stated on the card. The technical report repeats this table as Table 10. Effort levels unpublished; execution date unpublished.

81.7% · raw 81.7 %

Headline · unknown effort · Own vendor

Source/record date: 2026-06-04

NVIDIA Nemotron 3 Ultra BF16 card (revision 77df655d5e9f), benchmark table, row “IFBench (prompt loose)”. Collected with the NeMo Evaluator SDK (NeMo Gym, NeMo Skills and Harbor harnesses); pinned settings in NVIDIA’s reproducibility tutorial (https://github.com/NVIDIA-NeMo/Evaluator/blob/main/examples/nemotron/nemotron-3-ultra/reproducibility.md). The technical report repeats this table as Table 10. Effort levels unpublished; execution date unpublished.

63.8% · raw 63.8 %

Headline · unknown effort · Own vendor

Source/record date: 2026-06-04

NVIDIA Nemotron 3 Ultra BF16 card (revision 77df655d5e9f), benchmark table, row “Multi-Challenge”. Collected with the NeMo Evaluator SDK (NeMo Gym, NeMo Skills and Harbor harnesses); pinned settings in NVIDIA’s reproducibility tutorial (https://github.com/NVIDIA-NeMo/Evaluator/blob/main/examples/nemotron/nemotron-3-ultra/reproducibility.md). Scale AI Multi-Challenge packaged container. The technical report repeats this table as Table 10. Effort levels unpublished; execution date unpublished.

65.4% · raw 65.4 %

Headline · unknown effort · Own vendor

Source/record date: 2026-06-04

NVIDIA Nemotron 3 Ultra BF16 card (revision 77df655d5e9f), benchmark table, row “AA-LCR”. Collected with the NeMo Evaluator SDK (NeMo Gym, NeMo Skills and Harbor harnesses); pinned settings in NVIDIA’s reproducibility tutorial (https://github.com/NVIDIA-NeMo/Evaluator/blob/main/examples/nemotron/nemotron-3-ultra/reproducibility.md). Run by NVIDIA, not Artificial Analysis; version not stated. The technical report repeats this table as Table 10. Effort levels unpublished; execution date unpublished.

63.7% · raw 63.67 %

Alternative · unknown effort · Own vendor

Source/record date: 2026-06-04

NVIDIA technical report Table 17 (page 47), BF16 reference column, row “AA-LCR”: pass@1 63.67, mean completion length 4,110 tokens. A separate run from the card table, made to compare the NVFP4 checkpoint; the W4A16 and NVFP4 columns describe that other checkpoint and are not stored. Effort unpublished. Non-headline: the first-party row (65.4, https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16) holds this cell.

94.7% · raw 94.7 %

Headline · unknown effort · Own vendor

Source/record date: 2026-06-04

NVIDIA Nemotron 3 Ultra BF16 card (revision 77df655d5e9f), benchmark table, row “RULER (1M)”. Collected with the NeMo Evaluator SDK (NeMo Gym, NeMo Skills and Harbor harnesses); pinned settings in NVIDIA’s reproducibility tutorial (https://github.com/NVIDIA-NeMo/Evaluator/blob/main/examples/nemotron/nemotron-3-ultra/reproducibility.md). The technical report repeats this table as Table 10. Effort levels unpublished; execution date unpublished.

61.9% · raw 61.9 %

Headline · unknown effort · Own vendor

Source/record date: 2026-06-04

NVIDIA Nemotron 3 Ultra BF16 card (revision 77df655d5e9f), benchmark table, row “Longbench v2 (≤ 1M)”. Collected with the NeMo Evaluator SDK (NeMo Gym, NeMo Skills and Harbor harnesses); pinned settings in NVIDIA’s reproducibility tutorial (https://github.com/NVIDIA-NeMo/Evaluator/blob/main/examples/nemotron/nemotron-3-ultra/reproducibility.md). NVIDIA lists this benchmark among those run with the official implementation or an internal scaffold rather than its open-source NeMo tools. The technical report repeats this table as Table 10. Effort levels unpublished; execution date unpublished.

83% · raw 83 %

Headline · unknown effort · Own vendor

Source/record date: 2026-06-04

NVIDIA Nemotron 3 Ultra BF16 card (revision 77df655d5e9f), benchmark table, row “MMLU-ProX (avg en/de/fr/es/it/ja/zh/hi/pt/ko)”. Collected with the NeMo Evaluator SDK (NeMo Gym, NeMo Skills and Harbor harnesses); pinned settings in NVIDIA’s reproducibility tutorial (https://github.com/NVIDIA-NeMo/Evaluator/blob/main/examples/nemotron/nemotron-3-ultra/reproducibility.md). Mean over en/de/fr/es/it/ja/zh/hi/pt/ko. The technical report repeats this table as Table 10. Effort levels unpublished; execution date unpublished.

83.1% · raw 83.1 %

Alternative · unknown effort · Own vendor

Source/record date: 2026-10-15

NVIDIA’s own Nemotron 3 Ultra result in its later Nemotron 3.5 Super and Lightning technical report (15 October 2026), Table 10, column “N3-Ultra 550B-A55B”, row “MMLU-ProX”; the report does not state the precision. NeMo Gym; report Table 10 and §§3.8–3.9. Execution date and effort unpublished. Non-headline: the first-party row (83.0, https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16) holds this cell.

83.7 score · raw 83.7 score

Headline · unknown effort · Own vendor

Source/record date: 2026-06-04

NVIDIA Nemotron 3 Ultra BF16 card (revision 77df655d5e9f), benchmark table, row “WMT24++ (en→xx)”. Collected with the NeMo Evaluator SDK (NeMo Gym, NeMo Skills and Harbor harnesses); pinned settings in NVIDIA’s reproducibility tutorial (https://github.com/NVIDIA-NeMo/Evaluator/blob/main/examples/nemotron/nemotron-3-ultra/reproducibility.md). Scored with XCOMET-XXL (reproducibility notes); target languages not listed. The technical report repeats this table as Table 10. Effort levels unpublished; execution date unpublished.

82.3% · raw 82.3 %

Headline · unknown effort · Own vendor

Source/record date: 2026-06-04

NVIDIA technical report Table 11 (page 39): IMO-ProofBench Advanced accuracy 82.3% with graded score 173/210, human expert graders. Generate-verify-refine test-time scaling over several rounds (Figure 13 shows the IMO-ProofBench rounds), not a single pass. Precision of the run not restated; effort unpublished.

83.3% · raw 83.3 %

Headline · unknown effort · Own vendor

Source/record date: 2026-06-04

NVIDIA technical report Table 11 (page 39): IMO 2025 accuracy 83.3% with graded score 35/42, human expert graders. Generate-verify-refine test-time scaling over several rounds (Figure 13 shows the IMO-ProofBench rounds), not a single pass. Precision of the run not restated; effort unpublished.

96.7% · raw 96.7 %

Headline · unknown effort · Own vendor

Source/record date: 2026-06-04

NVIDIA technical report Table 11 (page 39): Putnam 2025 accuracy 96.7% with graded score 116/120, human expert graders. Generate-verify-refine test-time scaling over several rounds (Figure 13 shows the IMO-ProofBench rounds), not a single pass. Precision of the run not restated; effort unpublished.

97.6% · raw 97.6 %

Headline · unknown effort · Own vendor

Source/record date: 2026-06-04

NVIDIA technical report Table 11 (page 39): USAMO 2026 accuracy 97.6% with graded score 41/42, graded following Dekoninck et al. (2026). Generate-verify-refine test-time scaling over several rounds (Figure 13 shows the IMO-ProofBench rounds), not a single pass. Precision of the run not restated; effort unpublished.

82.1% · raw 82.12 %

Headline · unknown effort · Own vendor

Source/record date: 2026-06-04

NVIDIA technical report Table 17 (page 47), BF16 reference column, row “IFBench”: pass@1 82.12, mean completion length 5,272 tokens. A separate run from the card table, made to compare the NVFP4 checkpoint; the W4A16 and NVFP4 columns describe that other checkpoint and are not stored. IFBench variant (prompt or instruction, strict or loose) not stated in this table. Effort unpublished.

81.4% · raw 81.4 %

Alternative · unknown effort · Peer vendor

Source/record date: 2026-07-15

As reported by Thinking Machines in its Introducing Inkling post (15 July 2026), column “Nemotron 3 Ultra”, row “IFBench”. NVIDIA publishes no value for this exact benchmark, so this is Thinking Machines’ own run of the peer; precision, reasoning mode and effort for the peer are unpublished. Prompt or instruction level and strict or loose scoring unpublished. Not NVIDIA’s result and not an independent evaluator record. Non-headline: the first-party row (82.12, https://research.nvidia.com/labs/nemotron/files/NVIDIA-Nemotron-3-Ultra-Technical-Report.pdf#page=47) holds this cell.

67.3% · raw 67.3 %

Headline · unknown effort · Own vendor

Source/record date: 2026-06-04

NVIDIA technical report Figure 17 (page 65), agent × model matrix for SWE-bench Verified, row “ultra”, agent column “OpenCode (1.14.33)”: printed cell label read from the figure (crop in research-raw). The matrix’s average column (SWE-bench Verified 60.6, Terminal-Bench 2.1 48.9) is not stored. Agent-specific cell. Effort unpublished.

73.9% · raw 73.93 %

Alternative · unknown effort · Own vendor

Source/record date: 2026-10-15

NVIDIA’s own Nemotron 3 Ultra result in its later Nemotron 3.5 Super and Lightning technical report (15 October 2026), Table 10, column “N3-Ultra 550B-A55B”, row “SWE-benchVerified”; the report does not state the precision. NeMo Gym; report Table 10 and §§3.8–3.9. All SWE evaluations use OpenCode as the default harness. Execution date and effort unpublished. Non-headline: the first-party row (67.3, https://research.nvidia.com/labs/nemotron/files/NVIDIA-Nemotron-3-Ultra-Technical-Report.pdf#page=65) holds this cell.

70.3% · raw 70.3 %

Headline · unknown effort · Own vendor

Source/record date: 2026-06-04

NVIDIA technical report Figure 17 (page 65), agent × model matrix for SWE-bench Verified, row “ultra”, agent column “OpenHands (1.17.0)”: printed cell label read from the figure (crop in research-raw). The matrix’s average column (SWE-bench Verified 60.6, Terminal-Bench 2.1 48.9) is not stored. Agent-specific cell. Effort unpublished.

68.8% · raw 68.78 %

Headline · unknown effort · Own vendor

Source/record date: 2026-10-15

NVIDIA’s own Nemotron 3 Ultra result in its later Nemotron 3.5 Super and Lightning technical report (15 October 2026), Table 10, column “N3-Ultra 550B-A55B”, row “SWE-benchMultilingual”; the report does not state the precision. NeMo Gym; report Table 10 and §§3.8–3.9. All SWE evaluations use OpenCode as the default harness. Execution date and effort unpublished.

49.7% · raw 49.66 %

Headline · unknown effort · Own vendor

Source/record date: 2026-10-15

NVIDIA’s own Nemotron 3 Ultra result in its later Nemotron 3.5 Super and Lightning technical report (15 October 2026), Table 10, column “N3-Ultra 550B-A55B”, row “SWE-benchPro”; the report does not state the precision. NeMo Gym; report Table 10 and §§3.8–3.9. All SWE evaluations use OpenCode as the default harness. Execution date and effort unpublished.

73.7% · raw 73.68 %

Headline · unknown effort · Own vendor

Source/record date: 2026-10-15

NVIDIA’s own Nemotron 3 Ultra result in its later Nemotron 3.5 Super and Lightning technical report (15 October 2026), Table 10, column “N3-Ultra 550B-A55B”, row “Tau2Retail”; the report does not state the precision. NeMo Gym; report Table 10 and §§3.8–3.9. Execution date and effort unpublished.

79.8% · raw 79.75 %

Headline · unknown effort · Own vendor

Source/record date: 2026-10-15

NVIDIA’s own Nemotron 3 Ultra result in its later Nemotron 3.5 Super and Lightning technical report (15 October 2026), Table 10, column “N3-Ultra 550B-A55B”, row “Tau2Airline”; the report does not state the precision. NeMo Gym; report Table 10 and §§3.8–3.9. Execution date and effort unpublished.

83% · raw 83 %

Headline · unknown effort · Own vendor

Source/record date: 2026-10-15

NVIDIA’s own Nemotron 3 Ultra result in its later Nemotron 3.5 Super and Lightning technical report (15 October 2026), Table 10, column “N3-Ultra 550B-A55B”, row “Tau2Telecom”; the report does not state the precision. NeMo Gym; report Table 10 and §§3.8–3.9. Execution date and effort unpublished.

1148.3 · raw 1148.3 Elo

Headline · unknown effort · Own vendor

Source/record date: 2026-10-15

NVIDIA’s own Nemotron 3 Ultra result in its later Nemotron 3.5 Super and Lightning technical report (15 October 2026), Table 10, column “N3-Ultra 550B-A55B”, row “GDPVal-AA-v2ELO”; the report does not state the precision. NeMo Gym; report Table 10 and §§3.8–3.9. Stirrup agent, NVIDIA reference committee loosely following AA v2.1; not independent AA Elo. Execution date and effort unpublished.

32.4% · raw 32.4 %

Headline · unknown effort · Own vendor

Source/record date: 2026-10-15

NVIDIA’s own Nemotron 3 Ultra result in its later Nemotron 3.5 Super and Lightning technical report (15 October 2026), Table 10, column “N3-Ultra 550B-A55B”, row “GDPVal-AA-v2normELO”; the report does not state the precision. NeMo Gym; report Table 10 and §§3.8–3.9. Stirrup agent, NVIDIA reference committee loosely following AA v2.1; not independent AA Elo. Raw source normalized value ×100; not a pass rate. Execution date and effort unpublished.

67.9% · raw 67.85 %

Headline · unknown effort · Own vendor

Source/record date: 2026-10-15

NVIDIA’s own Nemotron 3 Ultra result in its later Nemotron 3.5 Super and Lightning technical report (15 October 2026), Table 10, column “N3-Ultra 550B-A55B”, row “BrowseComp”; the report does not state the precision. NeMo Gym; report Table 10 and §§3.8–3.9. NeMo Gym BrowseComp with Exa search. Execution date and effort unpublished.

61.1% · raw 61.06 %

Headline · unknown effort · Own vendor

Source/record date: 2026-10-15

NVIDIA’s own Nemotron 3 Ultra result in its later Nemotron 3.5 Super and Lightning technical report (15 October 2026), Table 10, column “N3-Ultra 550B-A55B”, row “BrowseComp(OpenCode)”; the report does not state the precision. NeMo Gym; report Table 10 and §§3.8–3.9. OpenCode harness. Execution date and effort unpublished.

13.5% · raw 13.46 %

Headline · unknown effort · Own vendor

Source/record date: 2026-10-15

NVIDIA’s own Nemotron 3 Ultra result in its later Nemotron 3.5 Super and Lightning technical report (15 October 2026), Table 10, column “N3-Ultra 550B-A55B”, row “SciCode(problem)”; the report does not state the precision. NeMo Gym; report Table 10 and §§3.8–3.9. SciCode uses scientist background / 8 repeats; whole-problem accuracy. Execution date and effort unpublished.

25.3% · raw 25.3 %

Headline · unknown effort · Own vendor

Source/record date: 2026-10-15

NVIDIA’s own Nemotron 3 Ultra result in its later Nemotron 3.5 Super and Lightning technical report (15 October 2026), Table 10, column “N3-Ultra 550B-A55B”, row “Accuracy”; the report does not state the precision. NeMo Gym; report Table 10 and §§3.8–3.9. AA Omniscience uses 600 public questions / 10 repeats with GPT-5.6 Luna judge. Execution date and effort unpublished.

77.9% · raw 77.91 %

Headline · unknown effort · Own vendor

Source/record date: 2026-10-15

NVIDIA’s own Nemotron 3 Ultra result in its later Nemotron 3.5 Super and Lightning technical report (15 October 2026), Table 10, column “N3-Ultra 550B-A55B”, row “Non-Hallucination”; the report does not state the precision. NeMo Gym; report Table 10 and §§3.8–3.9. AA Omniscience uses 600 public questions / 10 repeats with GPT-5.6 Luna judge. Execution date and effort unpublished.

75.2% · raw 75.19 %

Headline · unknown effort · Own vendor

Source/record date: 2026-10-15

NVIDIA’s own Nemotron 3 Ultra result in its later Nemotron 3.5 Super and Lightning technical report (15 October 2026), Table 10, column “N3-Ultra 550B-A55B”, row “AA-LCR”; the report does not state the precision. NeMo Gym; report Table 10 and §§3.8–3.9. Execution date and effort unpublished.

63.3% · raw 63.28 %

Headline · unknown effort · Own vendor

Source/record date: 2026-10-15

NVIDIA’s own Nemotron 3 Ultra result in its later Nemotron 3.5 Super and Lightning technical report (15 October 2026), Table 10, column “N3-Ultra 550B-A55B”, row “GraphWalks(≤1M)”; the report does not state the precision. NeMo Gym; report Table 10 and §§3.8–3.9. GraphWalks averages F1 across BFS and parents on the same 950/1,150 prompts fitting ≤1M. Execution date and effort unpublished.

82.9% · raw 82.94 %

Headline · unknown effort · Own vendor

Source/record date: 2026-10-15

NVIDIA’s own Nemotron 3 Ultra result in its later Nemotron 3.5 Super and Lightning technical report (15 October 2026), Table 10, column “N3-Ultra 550B-A55B”, row “IFBench”; the report does not state the precision. NeMo Gym; report Table 10 and §§3.8–3.9. IFBench is mean fraction of constraints satisfied under strict verification, not prompt all-pass. Execution date and effort unpublished.

34.3% · raw 34.27 %

Headline · unknown effort · Own vendor

Source/record date: 2026-10-15

NVIDIA’s own Nemotron 3 Ultra result in its later Nemotron 3.5 Super and Lightning technical report (15 October 2026), Table 10, column “N3-Ultra 550B-A55B”, row “NVChatBenchmark”; the report does not state the precision. NeMo Gym; report Table 10 and §§3.8–3.9. NVIDIA style-controlled chat preference win rate. Execution date and effort unpublished.

82.1 score · raw 82.06 score

Headline · unknown effort · Own vendor

Source/record date: 2026-10-15

NVIDIA’s own Nemotron 3 Ultra result in its later Nemotron 3.5 Super and Lightning technical report (15 October 2026), Table 10, column “N3-Ultra 550B-A55B”, row “WMT24ppCOMET”; the report does not state the precision. NeMo Gym; report Table 10 and §§3.8–3.9. XCOMET-XXL, English to nine languages. Execution date and effort unpublished.

82.7 score · raw 82.7 score

Alternative · unknown effort · Own vendor

Source/record date: 2026-10-15

NVIDIA’s own Nemotron 3 Ultra result in the Nemotron 3.5 Super VL BF16 card (15 October 2026), column “Nemotron 3 Ultra”; the card does not state the precision. Exact released BF16 card table. Conflicts/rounding versus detailed report remain non-headline alternatives; detailed report supplies the precise protocol. Raw card label: WMT24++. Execution date and effort unpublished. Non-headline: the first-party row (82.06, https://research.nvidia.com/labs/nemotron/files/NVIDIA-Nemotron-3.5-Super-and-Lightning-Technical-Report.pdf#page=23) holds this cell.

22.1% · raw 22.1 %

Headline · unknown effort · Own vendor

Source/record date: 2026-10-15

NVIDIA’s own Nemotron 3 Ultra result in the Nemotron 3.5 Super VL BF16 card (15 October 2026), column “Nemotron 3 Ultra”; the card does not state the precision. Exact released BF16 card table. Conflicts/rounding versus detailed report remain non-headline alternatives; detailed report supplies the precise protocol. Raw card label: AA-Omniscience hallucination rate. Execution date and effort unpublished.

8.8 · raw 8.8 index

Headline · unknown effort · Own vendor

Source/record date: 2026-10-15

NVIDIA’s own Nemotron 3 Ultra result in the Nemotron 3.5 Super VL BF16 card (15 October 2026), column “Nemotron 3 Ultra”; the card does not state the precision. Exact released BF16 card table. Conflicts/rounding versus detailed report remain non-headline alternatives; detailed report supplies the precise protocol. Raw card label: AA-Omniscience index. Execution date and effort unpublished.

8.6 · raw 8.6 index

Alternative · unknown effort · Peer vendor

Source/record date: 2026-10-05

Harness: Reflection launch evaluation; full agent, revision and token-budget settings not published. As reported by Reflection; peer comparison, not an independent evaluator record. Peer precision and effort unstated. Row “AA Omniscience (index) - public split only (our results)”, column “Nemotron 3 Ultra”. Reflection marks the public-split Omniscience row as its own results. Non-headline: the first-party row (8.8, https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-BF16) holds this cell.

665 score · raw 665 score

Headline · unknown effort · Own vendor

Source/record date: 2026-10-15

NVIDIA’s own Nemotron 3 Ultra result in the Nemotron 3.5 Super VL BF16 card (15 October 2026), column “Nemotron 3 Ultra”; the card does not state the precision. Exact card offline snapshot; full source text: 665 (836) | 866 (874) | 47. Tuple labels and parentheses are not explained; no quantity/denominator inferred. Reported external AA values are quotes, not independent verification. Execution date and effort unpublished.

866 score · raw 866 score

Headline · unknown effort · Own vendor

Source/record date: 2026-10-15

NVIDIA’s own Nemotron 3 Ultra result in the Nemotron 3.5 Super VL BF16 card (15 October 2026), column “Nemotron 3 Ultra”; the card does not state the precision. Exact card offline snapshot; full source text: 665 (836) | 866 (874) | 47. Tuple labels and parentheses are not explained; no quantity/denominator inferred. Reported external AA values are quotes, not independent verification. Execution date and effort unpublished.

47 score · raw 47 score

Headline · unknown effort · Own vendor

Source/record date: 2026-10-15

NVIDIA’s own Nemotron 3 Ultra result in the Nemotron 3.5 Super VL BF16 card (15 October 2026), column “Nemotron 3 Ultra”; the card does not state the precision. Exact card offline snapshot; full source text: 665 (836) | 866 (874) | 47. Tuple labels and parentheses are not explained; no quantity/denominator inferred. Reported external AA values are quotes, not independent verification. Execution date and effort unpublished.

23.3% · raw 23.28 %

Headline · unknown effort · Own vendor

Source/record date: 2026-10-15

NVIDIA’s own Nemotron 3 Ultra result in the Nemotron 3.5 Super VL BF16 card (15 October 2026), column “Nemotron 3 Ultra”; the card does not state the precision. Exact card offline snapshot; full source text: 23.28. Tuple labels and parentheses are not explained; no quantity/denominator inferred. Reported external AA values are quotes, not independent verification. Execution date and effort unpublished.

8.4% · raw 8.4 %

Headline · unknown effort · Own vendor

Source/record date: 2026-10-15

NVIDIA’s own Nemotron 3 Ultra result in the Nemotron 3.5 Super VL BF16 card (15 October 2026), column “Nemotron 3 Ultra”; the card does not state the precision. Exact card offline snapshot; full source text: 8.4 (AA 5.0). Tuple labels and parentheses are not explained; no quantity/denominator inferred. Reported external AA values are quotes, not independent verification. Execution date and effort unpublished.

0% · raw 0 %

Headline · unknown effort · Own vendor

Source/record date: 2026-10-15

NVIDIA’s own Nemotron 3 Ultra result in the Nemotron 3.5 Super VL BF16 card (15 October 2026), column “Nemotron 3 Ultra”; the card does not state the precision. Exact card offline snapshot; full source text: 0. Tuple labels and parentheses are not explained; no quantity/denominator inferred. Reported external AA values are quotes, not independent verification. Execution date and effort unpublished.

3% · raw 3 %

Headline · unknown effort · Own vendor

Source/record date: 2026-10-15

NVIDIA’s own Nemotron 3 Ultra result in the Nemotron 3.5 Super VL BF16 card (15 October 2026), column “Nemotron 3 Ultra”; the card does not state the precision. Exact card offline snapshot; full source text: 3. Tuple labels and parentheses are not explained; no quantity/denominator inferred. Reported external AA values are quotes, not independent verification. Execution date and effort unpublished.

94.2% · raw 94.2 %

Alternative · unknown effort · Peer vendor

Source/record date: 2026-07-15

As reported by Thinking Machines in its Introducing Inkling post (15 July 2026), column “Nemotron 3 Ultra”, row “AIME 2026”. NVIDIA publishes no value for this exact benchmark, so this is Thinking Machines’ own run of the peer; precision, reasoning mode and effort for the peer are unpublished. Sampling and repeats unpublished. Not NVIDIA’s result and not an independent evaluator record. Demoted 2026-11-05: duplicate of headline from https://huggingface.co/ricursiveAI/Ripple-65B-Preview/blob/240d5966cbe00f0627c699e59ef58560b60303cf/README.md (95.4); rule (c): all rows are peer comparisons and this source (2026-11-04) is later than the earlier one (2026-07-15).

95.4% · raw 95.4 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-11-04

Ricursive in-house run (italic in the Ripple-65B Preview card, revision 240d5966cbe0), row “AIME 2026”, column “Nemotron 3 Ultra 550B”; not a result from the model’s developer. Ricursive: all models tested at full BF16 precision with maximum effort levels; the peer’s exact effort name is unstated. Non-italic peer values in the table are the developers’ own figures and are not stored.

46.4% · raw 46.4 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-07-15

As reported by Thinking Machines in its Introducing Inkling post (15 July 2026), column “Nemotron 3 Ultra”, row “SWEBench Pro Public”. NVIDIA publishes no value for this exact benchmark, so this is Thinking Machines’ own run of the peer; precision, reasoning mode and effort for the peer are unpublished. Public set; harness unpublished. Not NVIDIA’s result and not an independent evaluator record.

32.4% · raw 32.4 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-07-15

As reported by Thinking Machines in its Introducing Inkling post (15 July 2026), column “Nemotron 3 Ultra”, row “SimpleQA Verified”. NVIDIA publishes no value for this exact benchmark, so this is Thinking Machines’ own run of the peer; precision, reasoning mode and effort for the peer are unpublished. Grader unpublished. Not NVIDIA’s result and not an independent evaluator record.

85.6% · raw 85.6 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-07-15

As reported by Thinking Machines in its Introducing Inkling post (15 July 2026), column “Nemotron 3 Ultra”, row “Global-MMLU-Lite”. NVIDIA publishes no value for this exact benchmark, so this is Thinking Machines’ own run of the peer; precision, reasoning mode and effort for the peer are unpublished. Prompting unpublished. Not NVIDIA’s result and not an independent evaluator record.

77.6% · raw 77.6 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-07-15

As reported by Thinking Machines in its Introducing Inkling post (15 July 2026), column “Nemotron 3 Ultra”, row “FORTRESS Adversarial”. NVIDIA publishes no value for this exact benchmark, so this is Thinking Machines’ own run of the peer; precision, reasoning mode and effort for the peer are unpublished. Higher is better: the rate of refusing harmful requests. Not NVIDIA’s result and not an independent evaluator record.

90.5% · raw 90.5 %

Alternative · unknown effort · Peer vendor

Source/record date: 2026-07-15

As reported by Thinking Machines in its Introducing Inkling post (15 July 2026), column “Nemotron 3 Ultra”, row “FORTRESS Benign”. NVIDIA publishes no value for this exact benchmark, so this is Thinking Machines’ own run of the peer; precision, reasoning mode and effort for the peer are unpublished. Higher is better: the rate of still answering safe look-alike requests. Not NVIDIA’s result and not an independent evaluator record. Demoted 2026-11-05: duplicate of headline from https://thinkingmachines.ai/news/inkling-small/ (90.6); rule (c): all rows are peer comparisons and this source (2026-07-30) is later than the earlier one (2026-07-15).

90.6% · raw 90.6 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-07-30

As reported by Thinking Machines in its Inkling-Small post (30 July 2026), column “Nemotron 3 Ultra”, row “FORTRESS Benign”. NVIDIA publishes no value for this exact setup, so this is Thinking Machines’ own run of the peer; precision, reasoning mode and effort for the peer are unpublished. Higher is better: the rate of still answering safe look-alike requests. Not NVIDIA’s result and not an independent evaluator record.

98.7% · raw 98.7 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-07-15

As reported by Thinking Machines in its Introducing Inkling post (15 July 2026), column “Nemotron 3 Ultra”, row “StrongREJECT”. NVIDIA publishes no value for this exact benchmark, so this is Thinking Machines’ own run of the peer; precision, reasoning mode and effort for the peer are unpublished. Higher is better. Not NVIDIA’s result and not an independent evaluator record.

78.8% · raw 78.8 %

Alternative · unknown effort · Peer vendor

Source/record date: 2026-07-30

As reported by Thinking Machines in its Inkling-Small post (30 July 2026), column “Nemotron 3 Ultra”, row “HMMT Feb 2026”. NVIDIA publishes no value for this exact setup, so this is Thinking Machines’ own run of the peer; precision, reasoning mode and effort for the peer are unpublished. Sampling and repeats unpublished. Not NVIDIA’s result and not an independent evaluator record. Demoted 2026-11-05: duplicate of headline from https://huggingface.co/ricursiveAI/Ripple-65B-Preview/blob/240d5966cbe00f0627c699e59ef58560b60303cf/README.md (80.3); rule (c): all rows are peer comparisons and this source (2026-11-04) is later than the earlier one (2026-07-30).

80.3% · raw 80.3 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-11-04

Ricursive in-house run (italic in the Ripple-65B Preview card, revision 240d5966cbe0), row “HMMT Feb 2026”, column “Nemotron 3 Ultra 550B”; not a result from the model’s developer. Ricursive: all models tested at full BF16 precision with maximum effort levels; the peer’s exact effort name is unstated. Non-italic peer values in the table are the developers’ own figures and are not stored.

92% · raw 92 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-11-04

Ricursive in-house run (italic in the Ripple-65B Preview card, revision 240d5966cbe0), row “MMLU”, column “Nemotron 3 Ultra 550B”; not a result from the model’s developer. Ricursive: all models tested at full BF16 precision with maximum effort levels; the peer’s exact effort name is unstated. Non-italic peer values in the table are the developers’ own figures and are not stored.

94.2% · raw 94.2 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-11-04

Ricursive in-house run (italic in the Ripple-65B Preview card, revision 240d5966cbe0), row “AIME 2024”, column “Nemotron 3 Ultra 550B”; not a result from the model’s developer. Ricursive: all models tested at full BF16 precision with maximum effort levels; the peer’s exact effort name is unstated. Non-italic peer values in the table are the developers’ own figures and are not stored.

94.2% · raw 94.2 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-11-04

Ricursive in-house run (italic in the Ripple-65B Preview card, revision 240d5966cbe0), row “AIME 2025”, column “Nemotron 3 Ultra 550B”; not a result from the model’s developer. Ricursive: all models tested at full BF16 precision with maximum effort levels; the peer’s exact effort name is unstated. Non-italic peer values in the table are the developers’ own figures and are not stored.

46.9% · raw 46.9 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-11-04

Ricursive in-house run (italic in the Ripple-65B Preview card, revision 240d5966cbe0), row “SWE-bench Pro v1.0”, column “Nemotron 3 Ultra 550B”; not a result from the model’s developer. Harness: OpenCode (the card says Ricursive uses OpenCode wherever no harness is named). Ricursive: all models tested at full BF16 precision with maximum effort levels; the peer’s exact effort name is unstated. Non-italic peer values in the table are the developers’ own figures and are not stored.

30.7% · raw 30.7 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-11-04

Ricursive in-house run (italic in the Ripple-65B Preview card, revision 240d5966cbe0), row “SWE-bench Pro v2.0 Hard”, column “Nemotron 3 Ultra 550B”; not a result from the model’s developer. Harness: OpenCode (the card says Ricursive uses OpenCode wherever no harness is named). The release blog’s tooltip for the same cell gives a separate “best harness” run. Ricursive: all models tested at full BF16 precision with maximum effort levels; the peer’s exact effort name is unstated. Non-italic peer values in the table are the developers’ own figures and are not stored.

0.2% · raw 0.2 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-11-04

Ricursive in-house run (italic in the Ripple-65B Preview card, revision 240d5966cbe0), row “DeepSWE 1.1”, column “Nemotron 3 Ultra 550B”; not a result from the model’s developer. Harness: OpenCode (the card says Ricursive uses OpenCode wherever no harness is named). The release blog’s tooltip for the same cell gives a separate “best harness” run. Ricursive: all models tested at full BF16 precision with maximum effort levels; the peer’s exact effort name is unstated. Non-italic peer values in the table are the developers’ own figures and are not stored.

0.6% · raw 0.6 %

Alternative · unknown effort · Peer vendor

Source/record date: 2026-11-04

Ricursive release blog bar chart “DeepSWE 1.1”, Nemotron 3 Ultra bar; tooltip “Best harness / Evaluated in-house” (read on 5 November 2026; receipt in research-raw). Not a result from the model’s developer. Harness: Ricursive’s best harness for this model, not named; the card’s 0.2 for the same cell is its OpenCode run. Ricursive: all models tested at full BF16 precision with maximum effort levels. Chart value, so the card’s table value keeps the cell. Non-headline: the spillover row (0.2, https://huggingface.co/ricursiveAI/Ripple-65B-Preview/blob/240d5966cbe00f0627c699e59ef58560b60303cf/README.md) holds this cell.

8.4% · raw 8.4 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-11-04

Ricursive in-house run (italic in the Ripple-65B Preview card, revision 240d5966cbe0), row “AutomationBench (Public)”, column “Nemotron 3 Ultra 550B”; not a result from the model’s developer. Harness: OpenCode (the card says Ricursive uses OpenCode wherever no harness is named). Ricursive: all models tested at full BF16 precision with maximum effort levels; the peer’s exact effort name is unstated. Non-italic peer values in the table are the developers’ own figures and are not stored.

62.2% · raw 62.2 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-11-04

Ricursive in-house run (italic in the Ripple-65B Preview card, revision 240d5966cbe0), row “MCP Atlas”, column “Nemotron 3 Ultra 550B”; not a result from the model’s developer. Harness: OpenCode (the card says Ricursive uses OpenCode wherever no harness is named). Ricursive: all models tested at full BF16 precision with maximum effort levels; the peer’s exact effort name is unstated. Non-italic peer values in the table are the developers’ own figures and are not stored.

79.3% · raw 79.3 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-11-04

Ricursive in-house run (italic in the Ripple-65B Preview card, revision 240d5966cbe0), row “FORTRESS Adversarial”, column “Nemotron 3 Ultra 550B”; not a result from the model’s developer. Ricursive: all models tested at full BF16 precision with maximum effort levels; the peer’s exact effort name is unstated. Non-italic peer values in the table are the developers’ own figures and are not stored.

89.1% · raw 89.1 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-11-04

Ricursive in-house run (italic in the Ripple-65B Preview card, revision 240d5966cbe0), row “FORTRESS Benign”, column “Nemotron 3 Ultra 550B”; not a result from the model’s developer. Ricursive: all models tested at full BF16 precision with maximum effort levels; the peer’s exact effort name is unstated. Non-italic peer values in the table are the developers’ own figures and are not stored.

98.2% · raw 98.2 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-11-04

Ricursive in-house run (italic in the Ripple-65B Preview card, revision 240d5966cbe0), row “StrongREJECT”, column “Nemotron 3 Ultra 550B”; not a result from the model’s developer. Ricursive: all models tested at full BF16 precision with maximum effort levels; the peer’s exact effort name is unstated. Non-italic peer values in the table are the developers’ own figures and are not stored.

80.7% · raw 80.7 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-15

Harness: Internal 202 behaviors, five attack conditions; mean union-broken ASRs with/without developer prompts; lower is better. Meta labels the peer “Nemotron 3 Ultra (thinking on)”; precision unstated. As reported by Meta in Muse Spark Open release, not direct evaluator verification. Source checked 15 October 2026; exact execution date unpublished.

72.8% · raw 72.8 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-17

As reported by IFM/Inception42 in the Jais-3-375B-A23B card (column “NVIDIA Nemotron 3 Ultra”), not an independent evaluation or NVIDIA’s own result. Same audited/subset protocol as Jais: Arabic suites use generation-based evaluation; starred source tests use audited cleaned question sets, not the unfiltered public suite. Exact source label: AraGen v3 (3C3H)09-2025. Peer precision and effort unstated.

86.1% · raw 86.1 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-17

As reported by IFM/Inception42 in the Jais-3-375B-A23B card (column “NVIDIA Nemotron 3 Ultra”), not an independent evaluation or NVIDIA’s own result. Same audited/subset protocol as Jais: Arabic suites use generation-based evaluation; starred source tests use audited cleaned question sets, not the unfiltered public suite. Exact source label: Correctness. Peer precision and effort unstated.

85.8% · raw 85.8 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-17

As reported by IFM/Inception42 in the Jais-3-375B-A23B card (column “NVIDIA Nemotron 3 Ultra”), not an independent evaluation or NVIDIA’s own result. Same audited/subset protocol as Jais: Arabic suites use generation-based evaluation; starred source tests use audited cleaned question sets, not the unfiltered public suite. Exact source label: Completeness. Peer precision and effort unstated.

16.6% · raw 16.6 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-17

As reported by IFM/Inception42 in the Jais-3-375B-A23B card (column “NVIDIA Nemotron 3 Ultra”), not an independent evaluation or NVIDIA’s own result. Same audited/subset protocol as Jais: Arabic suites use generation-based evaluation; starred source tests use audited cleaned question sets, not the unfiltered public suite. Exact source label: Conciseness. Peer precision and effort unstated.

77.4% · raw 77.4 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-17

As reported by IFM/Inception42 in the Jais-3-375B-A23B card (column “NVIDIA Nemotron 3 Ultra”), not an independent evaluation or NVIDIA’s own result. Same audited/subset protocol as Jais: Arabic suites use generation-based evaluation; starred source tests use audited cleaned question sets, not the unfiltered public suite. Exact source label: Helpfulness. Peer precision and effort unstated.

85% · raw 85 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-17

As reported by IFM/Inception42 in the Jais-3-375B-A23B card (column “NVIDIA Nemotron 3 Ultra”), not an independent evaluation or NVIDIA’s own result. Same audited/subset protocol as Jais: Arabic suites use generation-based evaluation; starred source tests use audited cleaned question sets, not the unfiltered public suite. Exact source label: Honesty. Peer precision and effort unstated.

85.8% · raw 85.8 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-17

As reported by IFM/Inception42 in the Jais-3-375B-A23B card (column “NVIDIA Nemotron 3 Ultra”), not an independent evaluation or NVIDIA’s own result. Same audited/subset protocol as Jais: Arabic suites use generation-based evaluation; starred source tests use audited cleaned question sets, not the unfiltered public suite. Exact source label: Harmlessness. Peer precision and effort unstated.

88.1% · raw 88.1 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-17

As reported by IFM/Inception42 in the Jais-3-375B-A23B card (column “NVIDIA Nemotron 3 Ultra”), not an independent evaluation or NVIDIA’s own result. Same audited/subset protocol as Jais: Arabic suites use generation-based evaluation; starred source tests use audited cleaned question sets, not the unfiltered public suite. Exact source label: Arabic MMLU*Accuracy. Audited subset: items with invalid options, ambiguous questions or indefensible gold labels removed. Peer precision and effort unstated.

93.1% · raw 93.1 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-17

As reported by IFM/Inception42 in the Jais-3-375B-A23B card (column “NVIDIA Nemotron 3 Ultra”), not an independent evaluation or NVIDIA’s own result. Same audited/subset protocol as Jais: Arabic suites use generation-based evaluation; starred source tests use audited cleaned question sets, not the unfiltered public suite. Exact source label: BelebeleAccuracy. Peer precision and effort unstated.

95.2% · raw 95.2 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-17

As reported by IFM/Inception42 in the Jais-3-375B-A23B card (column “NVIDIA Nemotron 3 Ultra”), not an independent evaluation or NVIDIA’s own result. Same audited/subset protocol as Jais: Arabic suites use generation-based evaluation; starred source tests use audited cleaned question sets, not the unfiltered public suite. Exact source label: AraSTEM*Accuracy. Audited subset: items with invalid options, ambiguous questions or indefensible gold labels removed. Peer precision and effort unstated.

93.7% · raw 93.7 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-17

As reported by IFM/Inception42 in the Jais-3-375B-A23B card (column “NVIDIA Nemotron 3 Ultra”), not an independent evaluation or NVIDIA’s own result. Same audited/subset protocol as Jais: Arabic suites use generation-based evaluation; starred source tests use audited cleaned question sets, not the unfiltered public suite. Exact source label: Arabic GSM8K*Accuracy. Audited subset: items with invalid options, ambiguous questions or indefensible gold labels removed. Peer precision and effort unstated.

98.4% · raw 98.4 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-17

As reported by IFM/Inception42 in the Jais-3-375B-A23B card (column “NVIDIA Nemotron 3 Ultra”), not an independent evaluation or NVIDIA’s own result. Same audited/subset protocol as Jais: Arabic suites use generation-based evaluation; starred source tests use audited cleaned question sets, not the unfiltered public suite. Exact source label: GATmath*Accuracy. Audited subset: items with invalid options, ambiguous questions or indefensible gold labels removed. Peer precision and effort unstated.

88.4% · raw 88.4 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-17

As reported by IFM/Inception42 in the Jais-3-375B-A23B card (column “NVIDIA Nemotron 3 Ultra”), not an independent evaluation or NVIDIA’s own result. Same audited/subset protocol as Jais: Arabic suites use generation-based evaluation; starred source tests use audited cleaned question sets, not the unfiltered public suite. Exact source label: MedAraBench*Accuracy. Audited subset: items with invalid options, ambiguous questions or indefensible gold labels removed. Peer precision and effort unstated.

77% · raw 77 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-17

As reported by IFM/Inception42 in the Jais-3-375B-A23B card (column “NVIDIA Nemotron 3 Ultra”), not an independent evaluation or NVIDIA’s own result. Same audited/subset protocol as Jais: Arabic suites use generation-based evaluation; starred source tests use audited cleaned question sets, not the unfiltered public suite. Exact source label: Arabic Legal (HELM)*Accuracy. Audited subset: items with invalid options, ambiguous questions or indefensible gold labels removed. Peer precision and effort unstated.

82.1% · raw 82.1 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-17

As reported by IFM/Inception42 in the Jais-3-375B-A23B card (column “NVIDIA Nemotron 3 Ultra”), not an independent evaluation or NVIDIA’s own result. Same audited/subset protocol as Jais: Arabic suites use generation-based evaluation; starred source tests use audited cleaned question sets, not the unfiltered public suite. Exact source label: IslamicQAAccuracy (mean of the 4 below). Peer precision and effort unstated.

93.5% · raw 93.5 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-17

As reported by IFM/Inception42 in the Jais-3-375B-A23B card (column “NVIDIA Nemotron 3 Ultra”), not an independent evaluation or NVIDIA’s own result. Same audited/subset protocol as Jais: Arabic suites use generation-based evaluation; starred source tests use audited cleaned question sets, not the unfiltered public suite. Exact source label: PaLM-X 2025-2*Accuracy. Audited subset: items with invalid options, ambiguous questions or indefensible gold labels removed. Peer precision and effort unstated.

83.6% · raw 83.6 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-17

As reported by IFM/Inception42 in the Jais-3-375B-A23B card (column “NVIDIA Nemotron 3 Ultra”), not an independent evaluation or NVIDIA’s own result. Same audited/subset protocol as Jais: Arabic suites use generation-based evaluation; starred source tests use audited cleaned question sets, not the unfiltered public suite. Exact source label: QIAS-1*Accuracy. Audited subset: items with invalid options, ambiguous questions or indefensible gold labels removed. Peer precision and effort unstated.

84.8% · raw 84.8 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-17

As reported by IFM/Inception42 in the Jais-3-375B-A23B card (column “NVIDIA Nemotron 3 Ultra”), not an independent evaluation or NVIDIA’s own result. Same audited/subset protocol as Jais: Arabic suites use generation-based evaluation; starred source tests use audited cleaned question sets, not the unfiltered public suite. Exact source label: QIAS-2*Accuracy. Audited subset: items with invalid options, ambiguous questions or indefensible gold labels removed. Peer precision and effort unstated.

66.7% · raw 66.7 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-17

As reported by IFM/Inception42 in the Jais-3-375B-A23B card (column “NVIDIA Nemotron 3 Ultra”), not an independent evaluation or NVIDIA’s own result. Same audited/subset protocol as Jais: Arabic suites use generation-based evaluation; starred source tests use audited cleaned question sets, not the unfiltered public suite. Exact source label: IslamicEval2025-1B*Accuracy. Audited subset: items with invalid options, ambiguous questions or indefensible gold labels removed. Peer precision and effort unstated.

87.9% · raw 87.9 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-17

As reported by IFM/Inception42 in the Jais-3-375B-A23B card (column “NVIDIA Nemotron 3 Ultra”), not an independent evaluation or NVIDIA’s own result. Same audited/subset protocol as Jais: Arabic suites use generation-based evaluation; starred source tests use audited cleaned question sets, not the unfiltered public suite. Exact source label: Arab CultureAccuracy (mean of the 3 below). Peer precision and effort unstated.

89.1% · raw 89.1 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-17

As reported by IFM/Inception42 in the Jais-3-375B-A23B card (column “NVIDIA Nemotron 3 Ultra”), not an independent evaluation or NVIDIA’s own result. Same audited/subset protocol as Jais: Arabic suites use generation-based evaluation; starred source tests use audited cleaned question sets, not the unfiltered public suite. Exact source label: ArabCulture*Accuracy. Audited subset: items with invalid options, ambiguous questions or indefensible gold labels removed. Peer precision and effort unstated.

88.8% · raw 88.8 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-17

As reported by IFM/Inception42 in the Jais-3-375B-A23B card (column “NVIDIA Nemotron 3 Ultra”), not an independent evaluation or NVIDIA’s own result. Same audited/subset protocol as Jais: Arabic suites use generation-based evaluation; starred source tests use audited cleaned question sets, not the unfiltered public suite. Exact source label: AraDiCE-Culture*Accuracy. Audited subset: items with invalid options, ambiguous questions or indefensible gold labels removed. Peer precision and effort unstated.

85.8% · raw 85.8 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-17

As reported by IFM/Inception42 in the Jais-3-375B-A23B card (column “NVIDIA Nemotron 3 Ultra”), not an independent evaluation or NVIDIA’s own result. Same audited/subset protocol as Jais: Arabic suites use generation-based evaluation; starred source tests use audited cleaned question sets, not the unfiltered public suite. Exact source label: Arabic Cuisine^Accuracy. Vendor says benchmark to be released; no public test-set availability assumed. Peer precision and effort unstated.

88.7% · raw 88.7 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-17

As reported by IFM/Inception42 in the Jais-3-375B-A23B card (column “NVIDIA Nemotron 3 Ultra”), not an independent evaluation or NVIDIA’s own result. Same audited/subset protocol as Jais: Arabic suites use generation-based evaluation; starred source tests use audited cleaned question sets, not the unfiltered public suite. Exact source label: Dialectal Arabic MMLU*Accuracy. Audited subset: items with invalid options, ambiguous questions or indefensible gold labels removed. Peer precision and effort unstated.

60.8% · raw 60.8 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-17

As reported by IFM/Inception42 in the Jais-3-375B-A23B card (column “NVIDIA Nemotron 3 Ultra”), not an independent evaluation or NVIDIA’s own result. Same audited/subset protocol as Jais: Arabic suites use generation-based evaluation; starred source tests use audited cleaned question sets, not the unfiltered public suite. Exact source label: Dialect Identification (QADI)Accuracy. Peer precision and effort unstated.

77.9% · raw 77.9 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-17

As reported by IFM/Inception42 in the Jais-3-375B-A23B card (column “NVIDIA Nemotron 3 Ultra”), not an independent evaluation or NVIDIA’s own result. Same audited/subset protocol as Jais: Arabic suites use generation-based evaluation; starred source tests use audited cleaned question sets, not the unfiltered public suite. Exact source label: Alyah*Accuracy. Audited subset: items with invalid options, ambiguous questions or indefensible gold labels removed. Peer precision and effort unstated.

87.4% · raw 87.4 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-17

As reported by IFM/Inception42 in the Jais-3-375B-A23B card (column “NVIDIA Nemotron 3 Ultra”), not an independent evaluation or NVIDIA’s own result. Same audited/subset protocol as Jais: Arabic suites use generation-based evaluation; starred source tests use audited cleaned question sets, not the unfiltered public suite. Exact source label: Arabic MMLU with diacriticsAccuracy. Peer precision and effort unstated.

69.7% · raw 69.7 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-17

As reported by IFM/Inception42 in the Jais-3-375B-A23B card (column “NVIDIA Nemotron 3 Ultra”), not an independent evaluation or NVIDIA’s own result. Same audited/subset protocol as Jais: Arabic suites use generation-based evaluation; starred source tests use audited cleaned question sets, not the unfiltered public suite. Exact source label: FannOrFlop-MCQAccuracy. Peer precision and effort unstated.

93.3% · raw 93.3 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-17

As reported by IFM/Inception42 in the Jais-3-375B-A23B card (column “NVIDIA Nemotron 3 Ultra”), not an independent evaluation or NVIDIA’s own result. Same audited/subset protocol as Jais: Arabic suites use generation-based evaluation; starred source tests use audited cleaned question sets, not the unfiltered public suite. Exact source label: Arabic Linguistics^Accuracy (mean of the 2 below). Vendor says benchmark to be released; no public test-set availability assumed. Peer precision and effort unstated.

92.3% · raw 92.3 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-17

As reported by IFM/Inception42 in the Jais-3-375B-A23B card (column “NVIDIA Nemotron 3 Ultra”), not an independent evaluation or NVIDIA’s own result. Same audited/subset protocol as Jais: Arabic suites use generation-based evaluation; starred source tests use audited cleaned question sets, not the unfiltered public suite. Exact source label: DiacritizationAccuracy. Peer precision and effort unstated.

94.3% · raw 94.3 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-17

As reported by IFM/Inception42 in the Jais-3-375B-A23B card (column “NVIDIA Nemotron 3 Ultra”), not an independent evaluation or NVIDIA’s own result. Same audited/subset protocol as Jais: Arabic suites use generation-based evaluation; starred source tests use audited cleaned question sets, not the unfiltered public suite. Exact source label: ParsingAccuracy. Peer precision and effort unstated.

83.9% · raw 83.9 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-17

As reported by IFM/Inception42 in the Jais-3-375B-A23B card (column “NVIDIA Nemotron 3 Ultra”), not an independent evaluation or NVIDIA’s own result. Same audited/subset protocol as Jais: Arabic suites use generation-based evaluation; starred source tests use audited cleaned question sets, not the unfiltered public suite. Exact source label: FLORES200+LLM-as-a-judge. Peer precision and effort unstated.

78.9% · raw 78.9 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-17

As reported by IFM/Inception42 in the Jais-3-375B-A23B card (column “NVIDIA Nemotron 3 Ultra”), not an independent evaluation or NVIDIA’s own result. Same audited/subset protocol as Jais: Arabic suites use generation-based evaluation; starred source tests use audited cleaned question sets, not the unfiltered public suite. Exact source label: Arabic IFBench^Accuracy (prompt-level strict). Vendor says benchmark to be released; no public test-set availability assumed. Peer precision and effort unstated.

79.2% · raw 79.2 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-17

As reported by IFM/Inception42 in the Jais-3-375B-A23B card (column “NVIDIA Nemotron 3 Ultra”), not an independent evaluation or NVIDIA’s own result. Same audited/subset protocol as Jais: Arabic suites use generation-based evaluation; starred source tests use audited cleaned question sets, not the unfiltered public suite. Exact source label: Arabic IFEvalAccuracy. Peer precision and effort unstated.

73% · raw 73 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-17

As reported by IFM/Inception42 in the Jais-3-375B-A23B card (column “NVIDIA Nemotron 3 Ultra”), not an independent evaluation or NVIDIA’s own result. Same audited/subset protocol as Jais: Arabic suites use generation-based evaluation; starred source tests use audited cleaned question sets, not the unfiltered public suite. Exact source label: AA-LCR (Ar)Accuracy. Peer precision and effort unstated.

71.2% · raw 71.2 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-17

As reported by IFM/Inception42 in the Jais-3-375B-A23B card (column “NVIDIA Nemotron 3 Ultra”), not an independent evaluation or NVIDIA’s own result. Same audited/subset protocol as Jais: Arabic suites use generation-based evaluation; starred source tests use audited cleaned question sets, not the unfiltered public suite. Exact source label: HalluScoreAccuracy. Peer precision and effort unstated.

22.9% · raw 22.9 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-17

As reported by IFM/Inception42 in the Jais-3-375B-A23B card (column “NVIDIA Nemotron 3 Ultra”), not an independent evaluation or NVIDIA’s own result. Same audited/subset protocol as Jais: Arabic suites use generation-based evaluation; starred source tests use audited cleaned question sets, not the unfiltered public suite. Exact source label: SimpleQA Verified (Ar)Aggregate (2/3 F-score + 1/3 non-hallucination rate). Aggregate = two-thirds F-score + one-third non-hallucination rate; not ordinary accuracy. Peer precision and effort unstated.

32.9% · raw 32.9 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-17

As reported by IFM/Inception42 in the Jais-3-375B-A23B card (column “NVIDIA Nemotron 3 Ultra”), not an independent evaluation or NVIDIA’s own result. Same audited/subset protocol as Jais: Arabic suites use generation-based evaluation; starred source tests use audited cleaned question sets, not the unfiltered public suite. Exact source label: F-score. Peer precision and effort unstated.

3% · raw 3 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-17

As reported by IFM/Inception42 in the Jais-3-375B-A23B card (column “NVIDIA Nemotron 3 Ultra”), not an independent evaluation or NVIDIA’s own result. Same audited/subset protocol as Jais: Arabic suites use generation-based evaluation; starred source tests use audited cleaned question sets, not the unfiltered public suite. Exact source label: Non-hallucination rate. Peer precision and effort unstated.

53.9% · raw 53.9 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-17

As reported by IFM/Inception42 in the Jais-3-375B-A23B card (column “NVIDIA Nemotron 3 Ultra”), not an independent evaluation or NVIDIA’s own result. Same audited/subset protocol as Jais: Arabic suites use generation-based evaluation; starred source tests use audited cleaned question sets, not the unfiltered public suite. Exact source label: Terminal Bench 2.1Accuracy (Avg@3). Accuracy Avg@3. Peer precision and effort unstated.

60% · raw 60 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-17

As reported by IFM/Inception42 in the Jais-3-375B-A23B card (column “NVIDIA Nemotron 3 Ultra”), not an independent evaluation or NVIDIA’s own result. Same audited/subset protocol as Jais: Arabic suites use generation-based evaluation; starred source tests use audited cleaned question sets, not the unfiltered public suite. Exact source label: GDPValAccuracy (@ 75% threshold). Task accuracy at a 75% rubric threshold, not Elo or mean rubric completion. Peer precision and effort unstated.

45.7% · raw 45.7 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-17

As reported by IFM/Inception42 in the Jais-3-375B-A23B card (column “NVIDIA Nemotron 3 Ultra”), not an independent evaluation or NVIDIA’s own result. Same audited/subset protocol as Jais: Arabic suites use generation-based evaluation; starred source tests use audited cleaned question sets, not the unfiltered public suite. Exact source label: MCPMarkAccuracy (Pass@1). Peer precision and effort unstated.

34.2% · raw 34.2 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-17

As reported by IFM/Inception42 in the Jais-3-375B-A23B card (column “NVIDIA Nemotron 3 Ultra”), not an independent evaluation or NVIDIA’s own result. Same audited/subset protocol as Jais: Arabic suites use generation-based evaluation; starred source tests use audited cleaned question sets, not the unfiltered public suite. Exact source label: WildClawBench (English text-only subset)Mean Score@3. English text-only subset, Mean Score@3; not full multimodal suite or pass@3. Peer precision and effort unstated.

9% · raw 9 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-17

As reported by IFM/Inception42 in the Jais-3-375B-A23B card (column “NVIDIA Nemotron 3 Ultra”), not an independent evaluation or NVIDIA’s own result. Same audited/subset protocol as Jais: Arabic suites use generation-based evaluation; starred source tests use audited cleaned question sets, not the unfiltered public suite. Exact source label: Apex-Agents (420 text-only subset, loop agent)Pass@1 (Accuracy). 420-task text-only subset, loop agent. Peer precision and effort unstated.

40% · raw 40 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-17

As reported by IFM/Inception42 in the Jais-3-375B-A23B card (column “NVIDIA Nemotron 3 Ultra”), not an independent evaluation or NVIDIA’s own result. Same audited/subset protocol as Jais: Arabic suites use generation-based evaluation; starred source tests use audited cleaned question sets, not the unfiltered public suite. Exact source label: GAIA v2 lit (Ar)Accuracy (Pass@1). Peer precision and effort unstated.

29.1% · raw 29.1 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-17

As reported by IFM/Inception42 in the Jais-3-375B-A23B card (column “NVIDIA Nemotron 3 Ultra”), not an independent evaluation or NVIDIA’s own result. Same audited/subset protocol as Jais: Arabic suites use generation-based evaluation; starred source tests use audited cleaned question sets, not the unfiltered public suite. Exact source label: Arabic File ProcessingAccuracy (@ 75% threshold). Task accuracy at a 75% rubric threshold, not Elo or mean rubric completion. Peer precision and effort unstated.

67.8% · raw 67.8 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-17

As reported by IFM/Inception42 in the Jais-3-375B-A23B card (column “NVIDIA Nemotron 3 Ultra”), not an independent evaluation or NVIDIA’s own result. Same audited/subset protocol as Jais: Arabic suites use generation-based evaluation; starred source tests use audited cleaned question sets, not the unfiltered public suite. Exact source label: SWE-Bench Verified (Offline)Accuracy (Pass@1). Offline protocol. Peer precision and effort unstated.

81.2% · raw 81.2 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-17

As reported by IFM/Inception42. Final-checkpoint launch summary; Arabic overall/domain averages weight benchmark components, not displayed subdomain averages. Agentic average covers nine suites, BrowseComp replaces Plus at flagship scale; those averages are not identical test pools. Card definitions preserve audited sets/thresholds/subsets. Column “Nemotron 3 Ultra”; peer precision and effort unstated.

90.6% · raw 90.6 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-17

As reported by IFM/Inception42. Final-checkpoint launch summary; Arabic overall/domain averages weight benchmark components, not displayed subdomain averages. Agentic average covers nine suites, BrowseComp replaces Plus at flagship scale; those averages are not identical test pools. Card definitions preserve audited sets/thresholds/subsets. Column “Nemotron 3 Ultra”; peer precision and effort unstated.

90.6% · raw 90.6 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-17

As reported by IFM/Inception42. Final-checkpoint launch summary; Arabic overall/domain averages weight benchmark components, not displayed subdomain averages. Agentic average covers nine suites, BrowseComp replaces Plus at flagship scale; those averages are not identical test pools. Card definitions preserve audited sets/thresholds/subsets. Column “Nemotron 3 Ultra”; peer precision and effort unstated.

95.8% · raw 95.8 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-17

As reported by IFM/Inception42. Final-checkpoint launch summary; Arabic overall/domain averages weight benchmark components, not displayed subdomain averages. Agentic average covers nine suites, BrowseComp replaces Plus at flagship scale; those averages are not identical test pools. Card definitions preserve audited sets/thresholds/subsets. Column “Nemotron 3 Ultra”; peer precision and effort unstated.

82.7% · raw 82.7 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-17

As reported by IFM/Inception42. Final-checkpoint launch summary; Arabic overall/domain averages weight benchmark components, not displayed subdomain averages. Agentic average covers nine suites, BrowseComp replaces Plus at flagship scale; those averages are not identical test pools. Card definitions preserve audited sets/thresholds/subsets. Column “Nemotron 3 Ultra”; peer precision and effort unstated.

82.5% · raw 82.5 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-17

As reported by IFM/Inception42. Final-checkpoint launch summary; Arabic overall/domain averages weight benchmark components, not displayed subdomain averages. Agentic average covers nine suites, BrowseComp replaces Plus at flagship scale; those averages are not identical test pools. Card definitions preserve audited sets/thresholds/subsets. Column “Nemotron 3 Ultra”; peer precision and effort unstated.

86.5% · raw 86.5 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-17

As reported by IFM/Inception42. Final-checkpoint launch summary; Arabic overall/domain averages weight benchmark components, not displayed subdomain averages. Agentic average covers nine suites, BrowseComp replaces Plus at flagship scale; those averages are not identical test pools. Card definitions preserve audited sets/thresholds/subsets. Column “Nemotron 3 Ultra”; peer precision and effort unstated.

75.8% · raw 75.8 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-17

As reported by IFM/Inception42. Final-checkpoint launch summary; Arabic overall/domain averages weight benchmark components, not displayed subdomain averages. Agentic average covers nine suites, BrowseComp replaces Plus at flagship scale; those averages are not identical test pools. Card definitions preserve audited sets/thresholds/subsets. Column “Nemotron 3 Ultra”; peer precision and effort unstated.

83.5% · raw 83.5 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-17

As reported by IFM/Inception42. Final-checkpoint launch summary; Arabic overall/domain averages weight benchmark components, not displayed subdomain averages. Agentic average covers nine suites, BrowseComp replaces Plus at flagship scale; those averages are not identical test pools. Card definitions preserve audited sets/thresholds/subsets. Column “Nemotron 3 Ultra”; peer precision and effort unstated.

65.1% · raw 65.1 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-17

As reported by IFM/Inception42. Final-checkpoint launch summary; Arabic overall/domain averages weight benchmark components, not displayed subdomain averages. Agentic average covers nine suites, BrowseComp replaces Plus at flagship scale; those averages are not identical test pools. Card definitions preserve audited sets/thresholds/subsets. Column “Nemotron 3 Ultra”; peer precision and effort unstated.

79.1% · raw 79.1 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-17

As reported by IFM/Inception42. Final-checkpoint launch summary; Arabic overall/domain averages weight benchmark components, not displayed subdomain averages. Agentic average covers nine suites, BrowseComp replaces Plus at flagship scale; those averages are not identical test pools. Card definitions preserve audited sets/thresholds/subsets. Column “Nemotron 3 Ultra”; peer precision and effort unstated.

47.1% · raw 47.1 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-17

As reported by IFM/Inception42. Final-checkpoint launch summary; Arabic overall/domain averages weight benchmark components, not displayed subdomain averages. Agentic average covers nine suites, BrowseComp replaces Plus at flagship scale; those averages are not identical test pools. Card definitions preserve audited sets/thresholds/subsets. Column “Nemotron 3 Ultra”; peer precision and effort unstated.

42.7% · raw 42.7 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-17

As reported by IFM/Inception42. Final-checkpoint launch summary; Arabic overall/domain averages weight benchmark components, not displayed subdomain averages. Agentic average covers nine suites, BrowseComp replaces Plus at flagship scale; those averages are not identical test pools. Card definitions preserve audited sets/thresholds/subsets. Column “Nemotron 3 Ultra”; peer precision and effort unstated.

85.6% · raw 85.6 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-16

Reflection Beam Safety Technical Report, 16 October release. Vendor-reported evaluation, including commissioned external work, not a directly verified independent board. Model execution dates and unstated precision/effort unknown. Source-specific population/harness; not a general safety guarantee. As reported by Reflection; exact peer label Nemotron 3 Ultra. Harm category “Abuse Entrapment & Coercive Control (“Grooming”)”.

67.1% · raw 67.1 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-16

Reflection Beam Safety Technical Report, 16 October release. Vendor-reported evaluation, including commissioned external work, not a directly verified independent board. Model execution dates and unstated precision/effort unknown. Source-specific population/harness; not a general safety guarantee. As reported by Reflection; exact peer label Nemotron 3 Ultra. Harm category “Biological Weapons”.

79.5% · raw 79.5 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-16

Reflection Beam Safety Technical Report, 16 October release. Vendor-reported evaluation, including commissioned external work, not a directly verified independent board. Model execution dates and unstated precision/effort unknown. Source-specific population/harness; not a general safety guarantee. As reported by Reflection; exact peer label Nemotron 3 Ultra. Harm category “Chemical Weapons”.

82.5% · raw 82.5 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-16

Reflection Beam Safety Technical Report, 16 October release. Vendor-reported evaluation, including commissioned external work, not a directly verified independent board. Model execution dates and unstated precision/effort unknown. Source-specific population/harness; not a general safety guarantee. As reported by Reflection; exact peer label Nemotron 3 Ultra. Harm category “Child Safety (Non-CSAM)”.

72.2% · raw 72.2 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-16

Reflection Beam Safety Technical Report, 16 October release. Vendor-reported evaluation, including commissioned external work, not a directly verified independent board. Model execution dates and unstated precision/effort unknown. Source-specific population/harness; not a general safety guarantee. As reported by Reflection; exact peer label Nemotron 3 Ultra. Harm category “Copyright, Protected-Text Recitation, and Memorized Source Continuation”.

82.8% · raw 82.8 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-16

Reflection Beam Safety Technical Report, 16 October release. Vendor-reported evaluation, including commissioned external work, not a directly verified independent board. Model execution dates and unstated precision/effort unknown. Source-specific population/harness; not a general safety guarantee. As reported by Reflection; exact peer label Nemotron 3 Ultra. Harm category “Cyber Offense”.

77.5% · raw 77.5 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-16

Reflection Beam Safety Technical Report, 16 October release. Vendor-reported evaluation, including commissioned external work, not a directly verified independent board. Model execution dates and unstated precision/effort unknown. Source-specific population/harness; not a general safety guarantee. As reported by Reflection; exact peer label Nemotron 3 Ultra. Harm category “Fraud & Deception”.

94% · raw 94 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-16

Reflection Beam Safety Technical Report, 16 October release. Vendor-reported evaluation, including commissioned external work, not a directly verified independent board. Model execution dates and unstated precision/effort unknown. Source-specific population/harness; not a general safety guarantee. As reported by Reflection; exact peer label Nemotron 3 Ultra. Harm category “Hate Speech & Harassment”.

81.5% · raw 81.5 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-16

Reflection Beam Safety Technical Report, 16 October release. Vendor-reported evaluation, including commissioned external work, not a directly verified independent board. Model execution dates and unstated precision/effort unknown. Source-specific population/harness; not a general safety guarantee. As reported by Reflection; exact peer label Nemotron 3 Ultra. Harm category “Illegal Activities”.

83.3% · raw 83.3 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-16

Reflection Beam Safety Technical Report, 16 October release. Vendor-reported evaluation, including commissioned external work, not a directly verified independent board. Model execution dates and unstated precision/effort unknown. Source-specific population/harness; not a general safety guarantee. As reported by Reflection; exact peer label Nemotron 3 Ultra. Harm category “Privacy Violations”.

68.7% · raw 68.7 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-16

Reflection Beam Safety Technical Report, 16 October release. Vendor-reported evaluation, including commissioned external work, not a directly verified independent board. Model execution dates and unstated precision/effort unknown. Source-specific population/harness; not a general safety guarantee. As reported by Reflection; exact peer label Nemotron 3 Ultra. Harm category “Radiological and Nuclear Weapons”.

82.7% · raw 82.7 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-16

Reflection Beam Safety Technical Report, 16 October release. Vendor-reported evaluation, including commissioned external work, not a directly verified independent board. Model execution dates and unstated precision/effort unknown. Source-specific population/harness; not a general safety guarantee. As reported by Reflection; exact peer label Nemotron 3 Ultra. Harm category “Self-Harm & Suicide”.

81.4% · raw 81.4 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-16

Reflection Beam Safety Technical Report, 16 October release. Vendor-reported evaluation, including commissioned external work, not a directly verified independent board. Model execution dates and unstated precision/effort unknown. Source-specific population/harness; not a general safety guarantee. As reported by Reflection; exact peer label Nemotron 3 Ultra. Harm category “Sex-Related Crimes (Non-CSAM)”.

92.2% · raw 92.2 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-16

Reflection Beam Safety Technical Report, 16 October release. Vendor-reported evaluation, including commissioned external work, not a directly verified independent board. Model execution dates and unstated precision/effort unknown. Source-specific population/harness; not a general safety guarantee. As reported by Reflection; exact peer label Nemotron 3 Ultra. Harm category “Sexual Content (Adult, Non-CSAM)”.

78.2% · raw 78.2 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-16

Reflection Beam Safety Technical Report, 16 October release. Vendor-reported evaluation, including commissioned external work, not a directly verified independent board. Model execution dates and unstated precision/effort unknown. Source-specific population/harness; not a general safety guarantee. As reported by Reflection; exact peer label Nemotron 3 Ultra. Harm category “Violence, Violent Crimes, Threats, and Incitement”.

2.7% · raw 2.7 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-16

Reflection Beam Safety Technical Report, 16 October release. Vendor-reported evaluation, including commissioned external work, not a directly verified independent board. Model execution dates and unstated precision/effort unknown. Source-specific population/harness; not a general safety guarantee. As reported by Reflection; exact peer label Nemotron 3 Ultra. Harmless-request behaviour breakdown.

1.1% · raw 1.1 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-16

Reflection Beam Safety Technical Report, 16 October release. Vendor-reported evaluation, including commissioned external work, not a directly verified independent board. Model execution dates and unstated precision/effort unknown. Source-specific population/harness; not a general safety guarantee. As reported by Reflection; exact peer label Nemotron 3 Ultra. Harmless-request behaviour breakdown.

1.5% · raw 1.5 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-16

Reflection Beam Safety Technical Report, 16 October release. Vendor-reported evaluation, including commissioned external work, not a directly verified independent board. Model execution dates and unstated precision/effort unknown. Source-specific population/harness; not a general safety guarantee. As reported by Reflection; exact peer label Nemotron 3 Ultra. Harmless-request behaviour breakdown.

94.6% · raw 94.6 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-16

Reflection Beam Safety Technical Report, 16 October release. Vendor-reported evaluation, including commissioned external work, not a directly verified independent board. Model execution dates and unstated precision/effort unknown. Source-specific population/harness; not a general safety guarantee. As reported by Reflection; exact peer label Nemotron 3 Ultra. Harmless-request behaviour breakdown.

7.3% · raw 7.3 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-16

Reflection Beam Safety Technical Report, 16 October release. Vendor-reported evaluation, including commissioned external work, not a directly verified independent board. Model execution dates and unstated precision/effort unknown. Source-specific population/harness; not a general safety guarantee. As reported by Reflection; exact peer label Nemotron 3 Ultra. Agentic harmful-compliance rate; lower is better.

44.1% · raw 44.1 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-16

Reflection Beam Safety Technical Report, 16 October release. Vendor-reported evaluation, including commissioned external work, not a directly verified independent board. Model execution dates and unstated precision/effort unknown. Source-specific population/harness; not a general safety guarantee. As reported by Reflection; exact peer label Nemotron 3 Ultra. Agentic harmful-compliance rate; lower is better.

13.6% · raw 13.61 %

Headline · unknown effort · Own vendor

Source/record date: 2026-06-04

Cognition primary public data.json (https://cognition.com/data/frontiercode-leaderboard/data.json, key v1_1), FrontierCode 1.1 main; weighted new_score x100, not unweighted correct. Harness: devin-cli. {"correct": 0.154, "tokens": 27744.3, "cost": 1.4655, "flagged_rate": 0.002}. Cost is source-observed USD per rollout; not an API rate. Data checked 5 November; evaluation date unpublished. Board effort label “none”; NVIDIA exposes reasoning on/off rather than effort levels, so the label is not mapped and effort stays unknown. Precision unstated.

27.2% · raw 27.21 %

Headline · unknown effort · Own vendor

Source/record date: 2026-06-04

Cognition primary public data.json (https://cognition.com/data/frontiercode-leaderboard/data.json, key v1_1), FrontierCode 1.1 extended; weighted new_score x100, not unweighted correct. Harness: devin-cli. {"correct": 0.3027, "tokens": 25833.2, "cost": 1.3168, "flagged_rate": 0.0013}. Cost is source-observed USD per rollout; not an API rate. Data checked 5 November; evaluation date unpublished. Board effort label “none”; NVIDIA exposes reasoning on/off rather than effort levels, so the label is not mapped and effort stays unknown. Precision unstated.

16.1% · raw 16.1 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-27

Harness: Mirendil-run OpenCode peer comparison of Nemotron 3 Ultra (OpenCode). Exact effort, checkpoint, precision, output budget and execution date unpublished. As reported by Mirendil in the Fern launch, figure “Agentic coding and terminal benchmarks”, row “SWE Atlas Codebase QnA”; not the peer vendor’s or an independent run.

31.4% · raw 31.4 %

Headline · unknown effort · Peer vendor

Source/record date: 2026-10-27

Harness: Mirendil-run OpenCode peer comparison of Nemotron 3 Ultra (OpenCode). Exact effort, checkpoint, precision, output budget and execution date unpublished. As reported by Mirendil in the Fern launch, figure “Agentic coding and terminal benchmarks”, row “SWE-Bench Pro v2-Hard”; not the peer vendor’s or an independent run.

6.3% · raw 6.3 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-02

pass^5, the share of 80 tasks solved on all 5 attempts; not pass@1. AA's "Nemotron 3 Ultra" entry (slug nvidia-nemotron-3-ultra-550b-a55b); AA does not state the precision or endpoint it used, so this is stored on the BF16 checkpoint entry, as for Nemotron 3.5 Super VL. AA's label states no reasoning effort. Manual observation of Artificial Analysis's public AA-AnalystAgent evaluation page, "AA-AnalystAgent: Score" chart, 2 November 2026 (all 51 listed configurations selected; rendered bar label paired with the x-axis model link; research-raw/aa-analyst-mlcr-2026-11-02/cap_analystagent_score.txt). Configuration "Nemotron 3 Ultra". AA's displayed rounding; no API, download, hidden precision or inferred run date. Added 2026-11-05 when the model was added to Examenos.

5.6% · raw 5.6 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-02

Pass rate on AA's private held-out expert and compound set, 60 questions run 3 times each. AA's "Nemotron 3 Ultra" entry (slug nvidia-nemotron-3-ultra-550b-a55b); AA does not state the precision or endpoint it used, so this is stored on the BF16 checkpoint entry, as for Nemotron 3.5 Super VL. AA's label states no reasoning effort. Manual observation of Artificial Analysis's public MLCR-AA evaluation page, "MLCR-AA: Score" chart, 2 November 2026 (all 176 listed configurations selected; rendered bar label paired with the x-axis model link; research-raw/aa-analyst-mlcr-2026-11-02/cap_mlcr_score.txt). Configuration "Nemotron 3 Ultra". AA's displayed rounding; no API, download, hidden precision or inferred run date. Added 2026-11-05 when the model was added to Examenos.

12 · raw 12 index

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Intelligence Index v5.0.1 (AA changelog, 3 November 2026: v5.0 with CritPt v2 regraded); never compared with v5.0 or v4.3.2 rows. AA's "Nemotron 3 Ultra" (slug nvidia-nemotron-3-ultra-550b-a55b). AA's label states no reasoning effort. Manual observation of Artificial Analysis's public Intelligence Index evaluation page (title "Artificial Analysis Intelligence Index v5.0.1"), "Artificial Analysis Intelligence Index: Score" chart, 5 November 2026 (106 configurations selected with ?models=; rendered bar label paired with the x-axis model link; research-raw/aa-v5.0.1-2026-11-05/cap_index_score.txt). Configuration "Nemotron 3 Ultra". AA's displayed rounding; no API, download, hidden precision or inferred run date.

$2.23/task · raw 2.23 USD/task

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

AA usage-priced cost per task at its list pricing; not an API tariff or invoice. Observed on the v5.0.1 page; bound to Intelligence Index v5.0.1 and never compared with v5.0 rows (five of the stored v5.0 cost and token values differ on the v5.0.1 page). AA's "Nemotron 3 Ultra" (slug nvidia-nemotron-3-ultra-550b-a55b). AA's label states no reasoning effort. Manual observation of Artificial Analysis's public Intelligence Index v5.0.1 evaluation page, "Cost per Intelligence Index Task" stacked chart (total label), 5 November 2026 (106 configurations selected with ?models=; rendered bar label paired with the x-axis model link; research-raw/aa-v5.0.1-2026-11-05/cap_index_cost.txt). Configuration "Nemotron 3 Ultra". AA's displayed rounding; no API, download, hidden precision or inferred run date.

51 k tokens/task · raw 51 k tokens/task

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Thousands of output tokens per task as displayed. Observed on the v5.0.1 page; bound to Intelligence Index v5.0.1 and never compared with v5.0 rows (five of the stored v5.0 cost and token values differ on the v5.0.1 page). AA's "Nemotron 3 Ultra" (slug nvidia-nemotron-3-ultra-550b-a55b). AA's label states no reasoning effort. Manual observation of Artificial Analysis's public Intelligence Index v5.0.1 evaluation page, "Output Tokens per Task" stacked chart (reasoning plus answer total label), 5 November 2026 (106 configurations selected with ?models=; rendered bar label paired with the x-axis model link; research-raw/aa-v5.0.1-2026-11-05/cap_index_tokens.txt). Configuration "Nemotron 3 Ultra". AA's displayed rounding; no API, download, hidden precision or inferred run date.

15% · raw 15 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

CritPt v2 hard subset as regraded by AA on 3 November 2026; not the full set or the earlier CritPt (AA) rows. AA's "Nemotron 3 Ultra" (slug nvidia-nemotron-3-ultra-550b-a55b). AA's label states no reasoning effort. Manual observation of Artificial Analysis's public CritPt v2 evaluation page, "Hard Subset" view, after AA's 3 November 2026 regrade, 5 November 2026 (106 configurations selected with ?models=; rendered bar label paired with the x-axis model link; research-raw/aa-v5.0.1-2026-11-05/cap_critpt_v2_hard.txt). Configuration "Nemotron 3 Ultra". AA's displayed rounding; no API, download, hidden precision or inferred run date.

3% · raw 3 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

AA's "Nemotron 3 Ultra" (slug nvidia-nemotron-3-ultra-550b-a55b). AA's label states no reasoning effort. Manual observation of Artificial Analysis's public AutomationBench-AA evaluation page, Score chart, 5 November 2026 (106 configurations selected with ?models=; rendered bar label paired with the x-axis model link; research-raw/aa-v5.0.1-2026-11-05/cap_automationbench.txt). Configuration "Nemotron 3 Ultra". AA's displayed rounding; no API, download, hidden precision or inferred run date.

877 · raw 877 Elo

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

AA's "Nemotron 3 Ultra" (slug nvidia-nemotron-3-ultra-550b-a55b). AA's label states no reasoning effort. Manual observation of Artificial Analysis's public AA-Briefcase evaluation page, combined Elo chart, 5 November 2026 (106 configurations selected with ?models=; rendered bar label paired with the x-axis model link; research-raw/aa-v5.0.1-2026-11-05/cap_briefcase_elo.txt). Configuration "Nemotron 3 Ultra". AA's displayed rounding; no API, download, hidden precision or inferred run date.

831 · raw 831 Elo

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

AA's "Nemotron 3 Ultra" (slug nvidia-nemotron-3-ultra-550b-a55b). AA's label states no reasoning effort. Manual observation of Artificial Analysis's public AA-Briefcase evaluation page, "Analytical Quality & Presentation Elo" tab (Analytical Quality series), 5 November 2026 (106 configurations selected with ?models=; rendered bar label paired with the x-axis model link; research-raw/aa-v5.0.1-2026-11-05/cap_briefcase_quality_presentation.txt). Configuration "Nemotron 3 Ultra". AA's displayed rounding; no API, download, hidden precision or inferred run date.

880 · raw 880 Elo

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

AA's "Nemotron 3 Ultra" (slug nvidia-nemotron-3-ultra-550b-a55b). AA's label states no reasoning effort. Manual observation of Artificial Analysis's public AA-Briefcase evaluation page, "Analytical Quality & Presentation Elo" tab (Presentation series), 5 November 2026 (106 configurations selected with ?models=; rendered bar label paired with the x-axis model link; research-raw/aa-v5.0.1-2026-11-05/cap_briefcase_quality_presentation.txt). Configuration "Nemotron 3 Ultra". AA's displayed rounding; no API, download, hidden precision or inferred run date.

22.0% · raw 22 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

AA's "Nemotron 3 Ultra" (slug nvidia-nemotron-3-ultra-550b-a55b). AA's label states no reasoning effort. Manual observation of Artificial Analysis's public AA-Briefcase evaluation page, "Rubric Score (%)" tab, 5 November 2026 (106 configurations selected with ?models=; rendered bar label paired with the x-axis model link; research-raw/aa-v5.0.1-2026-11-05/cap_briefcase_rubric.txt). Configuration "Nemotron 3 Ultra". AA's displayed rounding; no API, download, hidden precision or inferred run date.

3 · raw 3 index

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Signed knowledge index; v1.1 (3,000 questions) is not comparable with v1.0 rows. AA's "Nemotron 3 Ultra" (slug nvidia-nemotron-3-ultra-550b-a55b). AA's label states no reasoning effort. Manual observation of Artificial Analysis's public AA-Omniscience evaluation page, "AA-Omniscience v1.1: Net Score" chart, 5 November 2026 (106 configurations selected with ?models=; rendered bar label paired with the x-axis model link; research-raw/aa-v5.0.1-2026-11-05/cap_omniscience.txt). Configuration "Nemotron 3 Ultra". AA's displayed rounding; no API, download, hidden precision or inferred run date.

21% · raw 21 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Not comparable with v1.0 rows. AA's "Nemotron 3 Ultra" (slug nvidia-nemotron-3-ultra-550b-a55b). AA's label states no reasoning effort. Manual observation of Artificial Analysis's public AA-Omniscience evaluation page, accuracy chart, 5 November 2026 (106 configurations selected with ?models=; rendered bar label paired with the x-axis model link; research-raw/aa-v5.0.1-2026-11-05/cap_omniscience.txt). Configuration "Nemotron 3 Ultra". AA's displayed rounding; no API, download, hidden precision or inferred run date.

23% · raw 23 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Lower is better; not comparable with v1.0 rows. AA's "Nemotron 3 Ultra" (slug nvidia-nemotron-3-ultra-550b-a55b). AA's label states no reasoning effort. Manual observation of Artificial Analysis's public AA-Omniscience evaluation page, hallucination rate chart, 5 November 2026 (106 configurations selected with ?models=; rendered bar label paired with the x-axis model link; research-raw/aa-v5.0.1-2026-11-05/cap_omniscience.txt). Configuration "Nemotron 3 Ultra". AA's displayed rounding; no API, download, hidden precision or inferred run date.

0% · raw 0 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

AA's "Nemotron 3 Ultra" (slug nvidia-nemotron-3-ultra-550b-a55b). AA's label states no reasoning effort. Manual observation of Artificial Analysis's public Terminal-Bench-Science 0.1 evaluation page, Score chart, 5 November 2026 (106 configurations selected with ?models=; rendered bar label paired with the x-axis model link; research-raw/aa-v5.0.1-2026-11-05/cap_tbscience.txt). Configuration "Nemotron 3 Ultra". AA's displayed rounding; no API, download, hidden precision or inferred run date.

1.1% · raw 1.1 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

AA harness; not Cognition board rows. AA's "Nemotron 3 Ultra" (slug nvidia-nemotron-3-ultra-550b-a55b). AA's label states no reasoning effort. Manual observation of Artificial Analysis's public FrontierCode 2.0 evaluation page, Score chart, 5 November 2026 (106 configurations selected with ?models=; rendered bar label paired with the x-axis model link; research-raw/aa-v5.0.1-2026-11-05/cap_frontiercode2.txt). Configuration "Nemotron 3 Ultra". AA's displayed rounding; no API, download, hidden precision or inferred run date.

8.5% · raw 8.5 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Not Humanity's Last Exam (AA). AA's "Nemotron 3 Ultra" (slug nvidia-nemotron-3-ultra-550b-a55b). AA's label states no reasoning effort. Manual observation of Artificial Analysis's public HLE-Diamond evaluation page, Score chart, 5 November 2026 (106 configurations selected with ?models=; rendered bar label paired with the x-axis model link; research-raw/aa-v5.0.1-2026-11-05/cap_hle_diamond.txt). Configuration "Nemotron 3 Ultra". AA's displayed rounding; no API, download, hidden precision or inferred run date.

5% · raw 5 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

AA's "Nemotron 3 Ultra" (slug nvidia-nemotron-3-ultra-550b-a55b). AA's label states no reasoning effort. Manual observation of Artificial Analysis's public GDP.pdf evaluation page, "All-pass Rate" tab, 5 November 2026 (106 configurations selected with ?models=; rendered bar label paired with the x-axis model link; research-raw/aa-v5.0.1-2026-11-05/cap_gdppdf.txt). Configuration "Nemotron 3 Ultra". AA's displayed rounding; no API, download, hidden precision or inferred run date.

0.5% · raw 0.5 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

AA's "Nemotron 3 Ultra" (slug nvidia-nemotron-3-ultra-550b-a55b). AA's label states no reasoning effort. Manual observation of Artificial Analysis's public Terminal-Bench 4.0 evaluation page, Score chart, 5 November 2026 (106 configurations selected with ?models=; rendered bar label paired with the x-axis model link; research-raw/aa-v5.0.1-2026-11-05/cap_tb4.txt). Configuration "Nemotron 3 Ultra". AA's displayed rounding; no API, download, hidden precision or inferred run date.

135 tok/s · raw 135 tok/s

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

AA Performance Benchmarking v3.0.0 (3 November 2026): output tokens, including reasoning tokens, per second over the whole request including time to first token, averaged over the past 5 days for each workload (single-request and multi-turn agentic) and then across workloads; the model's first-party API, or the median across providers where there is none. A rolling observation; not the earlier "Output speed" rows. AA's "Nemotron 3 Ultra" (slug nvidia-nemotron-3-ultra-550b-a55b). AA's label states no reasoning effort. Manual observation of Artificial Analysis's public Muse Spark 1.3 model page, Performance section "Output Speed" chart (Performance Benchmarking v3.0.0; the same ?models= selection), 5 November 2026 (106 configurations selected with ?models=; rendered bar label paired with the x-axis model link; research-raw/aa-v5.0.1-2026-11-05/cap_output_speed_v3.txt). Configuration "Nemotron 3 Ultra". AA's displayed rounding; no API, download, hidden precision or inferred run date.

4.9% · raw 4.91 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Vals exact model page https://www.vals.ai/models/nvidia_nemotron-3-ultra-550b-a55b; source label Code Migration, 4.91%, published uncertainty ±1.59 percentage points; rank 73/79. Model page defaults are not inferred vendor product limits. Run execution date unpublished. Vals ran it via the NVIDIA API with top-p 0.95 (update dated 4 June 2026); reasoning effort and the endpoint’s precision are not stated. The update prose cites a 43.99% Vals Index of that date; the current page shows no Vals Index row, so none is stored.

31.7% · raw 31.73 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Vals exact model page https://www.vals.ai/models/nvidia_nemotron-3-ultra-550b-a55b; source label EMB, 31.73%, published uncertainty ±2.81 percentage points; rank 65/76. Model page defaults are not inferred vendor product limits. Run execution date unpublished. Vals ran it via the NVIDIA API with top-p 0.95 (update dated 4 June 2026); reasoning effort and the endpoint’s precision are not stated. The update prose cites a 43.99% Vals Index of that date; the current page shows no Vals Index row, so none is stored.

37.7% · raw 37.67 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Vals exact model page https://www.vals.ai/models/nvidia_nemotron-3-ultra-550b-a55b; source label Finance Agent (v2), 37.67%, published uncertainty ±0.15 percentage points; rank 68/81. Model page defaults are not inferred vendor product limits. Run execution date unpublished. Vals ran it via the NVIDIA API with top-p 0.95 (update dated 4 June 2026); reasoning effort and the endpoint’s precision are not stated. The update prose cites a 43.99% Vals Index of that date; the current page shows no Vals Index row, so none is stored.

15.380% · raw 15.38 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Vals exact model page https://www.vals.ai/models/nvidia_nemotron-3-ultra-550b-a55b; source label Legal Research Bench, 15.38%, published uncertainty ±2.51 percentage points; rank 74/90. Model page defaults are not inferred vendor product limits. Run execution date unpublished. Vals ran it via the NVIDIA API with top-p 0.95 (update dated 4 June 2026); reasoning effort and the endpoint’s precision are not stated. The update prose cites a 43.99% Vals Index of that date; the current page shows no Vals Index row, so none is stored.

7.640% · raw 7.64 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Vals exact model page https://www.vals.ai/models/nvidia_nemotron-3-ultra-550b-a55b; source label Vibe Code Bench v1.1, 7.64%, published uncertainty ±2.78 percentage points; rank 107/124. Model page defaults are not inferred vendor product limits. Run execution date unpublished. Harness: OpenHands. Vals ran it via the NVIDIA API with top-p 0.95 (update dated 4 June 2026); reasoning effort and the endpoint’s precision are not stated. The update prose cites a 43.99% Vals Index of that date; the current page shows no Vals Index row, so none is stored.

0% · raw 0 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Vals exact model page https://www.vals.ai/models/nvidia_nemotron-3-ultra-550b-a55b; source label ProgramBench, 0.00%, published uncertainty ±0.00 percentage points; rank 53/64. Model page defaults are not inferred vendor product limits. Run execution date unpublished. Vals ran it via the NVIDIA API with top-p 0.95 (update dated 4 June 2026); reasoning effort and the endpoint’s precision are not stated. The update prose cites a 43.99% Vals Index of that date; the current page shows no Vals Index row, so none is stored.

69% · raw 69 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Vals exact model page https://www.vals.ai/models/nvidia_nemotron-3-ultra-550b-a55b; source label SWE-bench, 69.00%, published uncertainty ±2.07 percentage points; rank 68/88. Model page defaults are not inferred vendor product limits. Run execution date unpublished. Vals ran it via the NVIDIA API with top-p 0.95 (update dated 4 June 2026); reasoning effort and the endpoint’s precision are not stated. The update prose cites a 43.99% Vals Index of that date; the current page shows no Vals Index row, so none is stored.

50.9% · raw 50.94 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Vals exact model page https://www.vals.ai/models/nvidia_nemotron-3-ultra-550b-a55b; source label Terminal-Bench 2.1, 50.94%, published uncertainty ±2.92 percentage points; rank 57/75. Model page defaults are not inferred vendor product limits. Run execution date unpublished. Vals ran it via the NVIDIA API with top-p 0.95 (update dated 4 June 2026); reasoning effort and the endpoint’s precision are not stated. The update prose cites a 43.99% Vals Index of that date; the current page shows no Vals Index row, so none is stored.

63.1% · raw 63.1 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: Scale MCP Atlas updated April 2026 method, maximum 100 tool calls/task, transient-error retry, updated LLM judge. Exact visible Nemotron 3 Ultra (thinking) pass rate 63.10%, 95% interval half-width 3.00 percentage points. Dataset 1000 tasks (500 public/500 heldout); page describes the public leaderboard subset as 500 tasks. 36 real MCP servers/220 tools. Footer update date does not establish this exact model execution date. Precision unstated.

34 correct · raw 34 correct

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-04

Harness: Plotly Studio Embedded, one run on 43 private English data-analytics questions in a synthetic wind-energy warehouse. Claude Sonnet 5 judges against Plotly answer key. One or two questions can be noise; no error bars. Exact checkpoint hash, run date and effort unpublished. Company-operated analytics benchmark, distinct from provider results. Table row as displayed: “Nemotron 3 Ultra† Jun 4, 2026 34 / 43 $7.06 21.0M 26 s / 1.7 min 1 h 12 min”; the dagger mark is not explained in the captured text. Serving precision unstated.

-0.158 IPS score · raw -0.1575690442802316 IPS score

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-01

Harness: Arena Agent Arena sessions with randomized component selection. Inverse-propensity-weighted estimate of the causal effect of this model choice on the aggregate of its user-outcome signals, relative to a uniform baseline over components, with time-decaying weights. Signed and relative to the published pool; zero is that baseline, not zero capability. 95% confidence interval -0.1904 to -0.1248; 216,338 observations; 13,536 sessions; rank 54 of 57 including unregistered models. Board published 28 October 2026 (dataset revision 323062f7, captured 1 November); a newer board may exist. Individual run dates and checkpoint hashes unpublished. Serving precision is not published on the board.

-0.221 IPS score · raw -0.22148931447425724 IPS score

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-01

Harness: Arena Agent Arena sessions with randomized component selection. Inverse-propensity-weighted estimate of the causal effect of this model choice on explicit user confirmation that the task was completed, relative to a uniform baseline over components, with time-decaying weights. Signed and relative to the published pool; zero is that baseline, not zero capability. 95% confidence interval -0.2999 to -0.1431; 4,206 observations; unpublished sessions; rank 54 of 57 including unregistered models. Board published 28 October 2026 (dataset revision 323062f7, captured 1 November); a newer board may exist. Individual run dates and checkpoint hashes unpublished. Serving precision is not published on the board.

-0.210 IPS score · raw -0.20965589007162178 IPS score

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-01

Harness: Arena Agent Arena sessions with randomized component selection. Inverse-propensity-weighted estimate of the causal effect of this model choice on tasks with more unprompted praise than complaints, relative to a uniform baseline over components, with time-decaying weights. Signed and relative to the published pool; zero is that baseline, not zero capability. 95% confidence interval -0.2795 to -0.1398; 1,246 observations; unpublished sessions; rank 53 of 57 including unregistered models. Board published 28 October 2026 (dataset revision 323062f7, captured 1 November); a newer board may exist. Individual run dates and checkpoint hashes unpublished. Serving precision is not published on the board.

-0.188 IPS score · raw -0.1878079431256432 IPS score

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-01

Harness: Arena Agent Arena sessions with randomized component selection. Inverse-propensity-weighted estimate of the causal effect of this model choice on less steering effort required from the user, relative to a uniform baseline over components, with time-decaying weights. Signed and relative to the published pool; zero is that baseline, not zero capability. 95% confidence interval -0.2535 to -0.1221; 6,787 observations; unpublished sessions; rank 55 of 57 including unregistered models. Board published 28 October 2026 (dataset revision 323062f7, captured 1 November); a newer board may exist. Individual run dates and checkpoint hashes unpublished. Serving precision is not published on the board.

-0.167 IPS score · raw -0.16726991369381777 IPS score

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-01

Harness: Arena Agent Arena sessions with randomized component selection. Inverse-propensity-weighted estimate of the causal effect of this model choice on fewer agent-caused bash failures per call, relative to a uniform baseline over components, with time-decaying weights. Signed and relative to the published pool; zero is that baseline, not zero capability. 95% confidence interval -0.2932 to -0.0414; 4,681 observations; unpublished sessions; rank 55 of 57 including unregistered models. Board published 28 October 2026 (dataset revision 323062f7, captured 1 November); a newer board may exist. Individual run dates and checkpoint hashes unpublished. Serving precision is not published on the board.

-0.006 IPS score · raw -0.006472594742611193 IPS score

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-01

Harness: Arena Agent Arena sessions with randomized component selection. Inverse-propensity-weighted estimate of the causal effect of this model choice on fewer hallucinated tool calls, relative to a uniform baseline over components, with time-decaying weights. Signed and relative to the published pool; zero is that baseline, not zero capability. 95% confidence interval -0.0170 to 0.0041; 199,418 observations; unpublished sessions; rank 41 of 57 including unregistered models. Board published 28 October 2026 (dataset revision 323062f7, captured 1 November); a newer board may exist. Individual run dates and checkpoint hashes unpublished. Serving precision is not published on the board.

611.366 s · raw 611.366 s

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: published Vals component protocol. Exact Vals task/version and source model. Requested effort, provider and diagnostics are retained with this observation; evaluation date and checkpoint hash are unpublished. Source update date is not execution date. Quality standard error is not a confidence interval. A source revision does not by itself establish a model update. Workload diagnostic; quality uncertainty does not describe its uncertainty. Not a token tariff or controlled efficiency ranking.

16.667% · raw 16.667 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: published Vals component protocol. Exact Vals task/version and source model. Requested effort, provider and diagnostics are retained with this observation; evaluation date and checkpoint hash are unpublished. Source update date is not execution date. Quality standard error is not a confidence interval. A source revision does not by itself establish a model update.

11.628% · raw 11.628 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: published Vals component protocol. Exact Vals task/version and source model. Requested effort, provider and diagnostics are retained with this observation; evaluation date and checkpoint hash are unpublished. Source update date is not execution date. Quality standard error is not a confidence interval. A source revision does not by itself establish a model update.

22.222% · raw 22.222 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: published Vals component protocol. Exact Vals task/version and source model. Requested effort, provider and diagnostics are retained with this observation; evaluation date and checkpoint hash are unpublished. Source update date is not execution date. Quality standard error is not a confidence interval. A source revision does not by itself establish a model update.

13.158% · raw 13.158 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: published Vals component protocol. Exact Vals task/version and source model. Requested effort, provider and diagnostics are retained with this observation; evaluation date and checkpoint hash are unpublished. Source update date is not execution date. Quality standard error is not a confidence interval. A source revision does not by itself establish a model update.

15.789% · raw 15.789 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: published Vals component protocol. Exact Vals task/version and source model. Requested effort, provider and diagnostics are retained with this observation; evaluation date and checkpoint hash are unpublished. Source update date is not execution date. Quality standard error is not a confidence interval. A source revision does not by itself establish a model update.

4.545% · raw 4.545 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: published Vals component protocol. Exact Vals task/version and source model. Requested effort, provider and diagnostics are retained with this observation; evaluation date and checkpoint hash are unpublished. Source update date is not execution date. Quality standard error is not a confidence interval. A source revision does not by itself establish a model update.

16.667% · raw 16.667 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: published Vals component protocol. Exact Vals task/version and source model. Requested effort, provider and diagnostics are retained with this observation; evaluation date and checkpoint hash are unpublished. Source update date is not execution date. Quality standard error is not a confidence interval. A source revision does not by itself establish a model update.

9.091% · raw 9.091 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: published Vals component protocol. Exact Vals task/version and source model. Requested effort, provider and diagnostics are retained with this observation; evaluation date and checkpoint hash are unpublished. Source update date is not execution date. Quality standard error is not a confidence interval. A source revision does not by itself establish a model update.

68.005% · raw 68.005 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: published Vals component protocol. Exact Vals task/version and source model. Requested effort, provider and diagnostics are retained with this observation; evaluation date and checkpoint hash are unpublished. Source update date is not execution date. Quality standard error is not a confidence interval. A source revision does not by itself establish a model update.

72.398% · raw 72.398 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: published Vals component protocol. Exact Vals task/version and source model. Requested effort, provider and diagnostics are retained with this observation; evaluation date and checkpoint hash are unpublished. Source update date is not execution date. Quality standard error is not a confidence interval. A source revision does not by itself establish a model update.

70.541% · raw 70.541 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: published Vals component protocol. Exact Vals task/version and source model. Requested effort, provider and diagnostics are retained with this observation; evaluation date and checkpoint hash are unpublished. Source update date is not execution date. Quality standard error is not a confidence interval. A source revision does not by itself establish a model update.

71.520% · raw 71.52 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: published Vals component protocol. Exact Vals task/version and source model. Requested effort, provider and diagnostics are retained with this observation; evaluation date and checkpoint hash are unpublished. Source update date is not execution date. Quality standard error is not a confidence interval. A source revision does not by itself establish a model update.

69.251% · raw 69.251 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: published Vals component protocol. Exact Vals task/version and source model. Requested effort, provider and diagnostics are retained with this observation; evaluation date and checkpoint hash are unpublished. Source update date is not execution date. Quality standard error is not a confidence interval. A source revision does not by itself establish a model update.

66.155% · raw 66.155 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: published Vals component protocol. Exact Vals task/version and source model. Requested effort, provider and diagnostics are retained with this observation; evaluation date and checkpoint hash are unpublished. Source update date is not execution date. Quality standard error is not a confidence interval. A source revision does not by itself establish a model update.

51.562% · raw 51.562 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: published Vals component protocol. Exact Vals task/version and source model. Requested effort, provider and diagnostics are retained with this observation; evaluation date and checkpoint hash are unpublished. Source update date is not execution date. Quality standard error is not a confidence interval. A source revision does not by itself establish a model update.

73.530% · raw 73.53 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: published Vals component protocol. Exact Vals task/version and source model. Requested effort, provider and diagnostics are retained with this observation; evaluation date and checkpoint hash are unpublished. Source update date is not execution date. Quality standard error is not a confidence interval. A source revision does not by itself establish a model update.

73.479% · raw 73.479 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: published Vals component protocol. Exact Vals task/version and source model. Requested effort, provider and diagnostics are retained with this observation; evaluation date and checkpoint hash are unpublished. Source update date is not execution date. Quality standard error is not a confidence interval. A source revision does not by itself establish a model update.

2217.462 s · raw 2217.462 s

Headline · unknown effort · Independent evaluator

Source/record date: 2026-11-05

Harness: OpenHands. Exact Vals task/version and source model. Requested effort, provider and diagnostics are retained with this observation; evaluation date and checkpoint hash are unpublished. Source update date is not execution date. Quality standard error is not a confidence interval. A source revision does not by itself establish a model update. Workload diagnostic; quality uncertainty does not describe its uncertainty. Not a token tariff or controlled efficiency ranking.