Examenos

81.9% · raw 81.94 %

Headline · unknown effort · Own vendor

Source/record date: 2026-10-19

Harness: NVIDIA release NeMo Gym/Evaluator; BF16 reference weights. MMLU Pro. Source comparison metrics are vendor-run, not independent AA. Thinking enabled without an API effort label; benchmark-specific sampling/repeats are in the official recipes.

17.5 score · raw 17.5 score

Headline · unknown effort · Own vendor

Source/record date: 2026-10-19

Harness: NVIDIA release NeMo Gym/Evaluator; BF16 reference weights. AA-Omniscience. Source comparison metrics are vendor-run, not independent AA. Thinking enabled without an API effort label; benchmark-specific sampling/repeats are in the official recipes. The broad source label does not establish the signed AA index versus accuracy quantity; preserve the native score, without ranking or substituting it for directly labeled report components.

75.4% · raw 75.44 %

Headline · unknown effort · Own vendor

Source/record date: 2026-10-19

Harness: NVIDIA release NeMo Gym/Evaluator; BF16 reference weights. GPQA Diamond (no tools). Source comparison metrics are vendor-run, not independent AA. Thinking enabled without an API effort label; benchmark-specific sampling/repeats are in the official recipes.

11.7% · raw 11.72 %

Headline · unknown effort · Own vendor

Source/record date: 2026-10-19

Harness: NVIDIA release NeMo Gym/Evaluator; BF16 reference weights. HLE (text-only, no tools). Source comparison metrics are vendor-run, not independent AA. Thinking enabled without an API effort label; benchmark-specific sampling/repeats are in the official recipes.

32.6% · raw 32.6 %

Headline · unknown effort · Own vendor

Source/record date: 2026-10-19

Harness: NVIDIA release NeMo Gym/Evaluator; BF16 reference weights. SciCode. Source comparison metrics are vendor-run, not independent AA. Thinking enabled without an API effort label; benchmark-specific sampling/repeats are in the official recipes.

51.6% · raw 51.56 %

Headline · unknown effort · Own vendor

Source/record date: 2026-10-19

Harness: NVIDIA release NeMo Gym/Evaluator; BF16 reference weights. SWE-bench Verified. Source comparison metrics are vendor-run, not independent AA. Thinking enabled without an API effort label; benchmark-specific sampling/repeats are in the official recipes.

39.3% · raw 39.33 %

Headline · unknown effort · Own vendor

Source/record date: 2026-10-19

Harness: NVIDIA release NeMo Gym/Evaluator; BF16 reference weights. SWE-bench Multilingual. Source comparison metrics are vendor-run, not independent AA. Thinking enabled without an API effort label; benchmark-specific sampling/repeats are in the official recipes.

24.6% · raw 24.58 %

Headline · unknown effort · Own vendor

Source/record date: 2026-10-19

Harness: NVIDIA release NeMo Gym/Evaluator; BF16 reference weights. Terminal-Bench 2.1. Source comparison metrics are vendor-run, not independent AA. Thinking enabled without an API effort label; benchmark-specific sampling/repeats are in the official recipes.

85.4% · raw 85.37 %

Headline · unknown effort · Own vendor

Source/record date: 2026-10-19

Harness: NVIDIA release NeMo Gym/Evaluator; BF16 reference weights. PinchBench. Source comparison metrics are vendor-run, not independent AA. Thinking enabled without an API effort label; benchmark-specific sampling/repeats are in the official recipes.

37% · raw 36.97 %

Headline · unknown effort · Own vendor

Source/record date: 2026-10-19

Harness: NVIDIA release NeMo Gym/Evaluator; BF16 reference weights. BrowseComp. Source comparison metrics are vendor-run, not independent AA. Thinking enabled without an API effort label; benchmark-specific sampling/repeats are in the official recipes.

9.3% · raw 9.28 %

Headline · unknown effort · Own vendor

Source/record date: 2026-10-19

Harness: NVIDIA release NeMo Gym/Evaluator; BF16 reference weights. τ³-bench (Banking). Source comparison metrics are vendor-run, not independent AA. Thinking enabled without an API effort label; benchmark-specific sampling/repeats are in the official recipes.

832 · raw 832 Elo

Headline · unknown effort · Own vendor

Source/record date: 2026-10-19

Harness: NVIDIA release NeMo Gym/Evaluator; BF16 reference weights. GDPval-AA-V2. Source comparison metrics are vendor-run, not independent AA. Thinking enabled without an API effort label; benchmark-specific sampling/repeats are in the official recipes.

71.9% · raw 71.88 %

Headline · unknown effort · Own vendor

Source/record date: 2026-10-19

Harness: NVIDIA release NeMo Gym/Evaluator; BF16 reference weights. IFBench (loose). Source comparison metrics are vendor-run, not independent AA. Thinking enabled without an API effort label; benchmark-specific sampling/repeats are in the official recipes.

52% · raw 52 %

Headline · unknown effort · Own vendor

Source/record date: 2026-10-19

Harness: NVIDIA release NeMo Gym/Evaluator; BF16 reference weights. AA-LCR. Source comparison metrics are vendor-run, not independent AA. Thinking enabled without an API effort label; benchmark-specific sampling/repeats are in the official recipes.

60% · raw 60 %

Headline · unknown effort · Own vendor

Source/record date: 2026-10-19

Harness: OpenCode v1.17.8. NVIDIA source figure, BF16 card; 500 tasks. Exact effort/budget and execution date unpublished. No best-across-harness substitution.

58.4% · raw 58.4 %

Headline · unknown effort · Own vendor

Source/record date: 2026-10-19

Harness: Copilot v1.0.71. NVIDIA source figure, BF16 card; 500 tasks. Exact effort/budget and execution date unpublished. No best-across-harness substitution.

57.4% · raw 57.4 %

Headline · unknown effort · Own vendor

Source/record date: 2026-10-19

Harness: Claude v2.1.126. NVIDIA source figure, BF16 card; 500 tasks. Exact effort/budget and execution date unpublished. No best-across-harness substitution.

54.5% · raw 54.5 %

Headline · unknown effort · Own vendor

Source/record date: 2026-10-19

Harness: Pi v0.80.10. NVIDIA source figure, BF16 card; 500 tasks. Exact effort/budget and execution date unpublished. No best-across-harness substitution.

50.7% · raw 50.7 %

Headline · unknown effort · Own vendor

Source/record date: 2026-10-19

Harness: Mini-SWE agent v2.4.5. NVIDIA source figure, BF16 card; 500 tasks. Exact effort/budget and execution date unpublished. No best-across-harness substitution.

48.7% · raw 48.7 %

Headline · unknown effort · Own vendor

Source/record date: 2026-10-19

Harness: OpenHands v1.36.1. NVIDIA source figure, BF16 card; 500 tasks. Exact effort/budget and execution date unpublished. No best-across-harness substitution.

45.7% · raw 45.7 %

Headline · unknown effort · Own vendor

Source/record date: 2026-10-19

Harness: Hermes v0.18.2. NVIDIA source figure, BF16 card; 500 tasks. Exact effort/budget and execution date unpublished. No best-across-harness substitution.

11% · raw 11 %

Headline · unknown effort · Own vendor

Source/record date: 2026-10-19

Harness: Codex v0.144.6. NVIDIA source figure, BF16 card; 500 tasks. Exact effort/budget and execution date unpublished. No best-across-harness substitution.

29.7% · raw 29.7 %

Headline · unknown effort · Own vendor

Source/record date: 2026-10-19

Harness: Mini-SWE agent v2.4.5. NVIDIA source figure, BF16 card; 89 tasks. Exact effort/budget and execution date unpublished. No best-across-harness substitution.

29.1% · raw 29.1 %

Headline · unknown effort · Own vendor

Source/record date: 2026-10-19

Harness: OpenCode v1.17.8. NVIDIA source figure, BF16 card; 89 tasks. Exact effort/budget and execution date unpublished. No best-across-harness substitution.

29% · raw 29 %

Headline · unknown effort · Own vendor

Source/record date: 2026-10-19

Harness: Claude v2.1.126. NVIDIA source figure, BF16 card; 89 tasks. Exact effort/budget and execution date unpublished. No best-across-harness substitution.

27% · raw 27 %

Headline · unknown effort · Own vendor

Source/record date: 2026-10-19

Harness: Copilot v1.0.71. NVIDIA source figure, BF16 card; 89 tasks. Exact effort/budget and execution date unpublished. No best-across-harness substitution.

26.1% · raw 26.1 %

Headline · unknown effort · Own vendor

Source/record date: 2026-10-19

Harness: OpenHands v1.36.1. NVIDIA source figure, BF16 card; 89 tasks. Exact effort/budget and execution date unpublished. No best-across-harness substitution.

25.8% · raw 25.8 %

Headline · unknown effort · Own vendor

Source/record date: 2026-10-19

Harness: Hermes v0.18.2. NVIDIA source figure, BF16 card; 89 tasks. Exact effort/budget and execution date unpublished. No best-across-harness substitution.

24.5% · raw 24.5 %

Headline · unknown effort · Own vendor

Source/record date: 2026-10-19

Harness: Pi v0.80.10. NVIDIA source figure, BF16 card; 89 tasks. Exact effort/budget and execution date unpublished. No best-across-harness substitution.

2.9% · raw 2.9 %

Headline · unknown effort · Own vendor

Source/record date: 2026-10-19

Harness: Codex v0.144.6. NVIDIA source figure, BF16 card; 89 tasks. Exact effort/budget and execution date unpublished. No best-across-harness substitution.

52.2% · raw 52.2 %

Headline · unknown effort · Own vendor

Source/record date: 2026-10-19

Harness: NVIDIA NeMo Gym/Evaluator, report Table22/§4.9. BF16 target, FP8 KV and FP32 Mamba SSM cache. Published saved-view result; card differences are explicitly attributed to stochastic/judge variation, not a new checkpoint. Printed precision retained, no recovery/delta recomputation. Quantized NVFP4 columns are separate unapproved reference variants and are not copied as BF16. Native published percent. CritPt zero is a genuine reported result; recovery is undefined, never zero. GDPval uses normalized committee Elo, not raw Elo.

24569 tokens/trajectory · raw 24569 tokens/trajectory

Headline · unknown effort · Own vendor

Source/record date: 2026-10-19

Harness: NVIDIA NeMo Gym/Evaluator, report Table22/§4.9. BF16 target, FP8 KV and FP32 Mamba SSM cache. Published saved-view result; card differences are explicitly attributed to stochastic/judge variation, not a new checkpoint. Printed precision retained, no recovery/delta recomputation. Quantized NVFP4 columns are separate unapproved reference variants and are not copied as BF16. Native mean output tokens per trajectory.

37.1% · raw 37.13 %

Headline · unknown effort · Own vendor

Source/record date: 2026-10-19

Harness: NVIDIA NeMo Gym/Evaluator, report Table22/§4.9. BF16 target, FP8 KV and FP32 Mamba SSM cache. Published saved-view result; card differences are explicitly attributed to stochastic/judge variation, not a new checkpoint. Printed precision retained, no recovery/delta recomputation. Quantized NVFP4 columns are separate unapproved reference variants and are not copied as BF16. Native published percent. CritPt zero is a genuine reported result; recovery is undefined, never zero. GDPval uses normalized committee Elo, not raw Elo.

31071 tokens/trajectory · raw 31071 tokens/trajectory

Headline · unknown effort · Own vendor

Source/record date: 2026-10-19

Harness: NVIDIA NeMo Gym/Evaluator, report Table22/§4.9. BF16 target, FP8 KV and FP32 Mamba SSM cache. Published saved-view result; card differences are explicitly attributed to stochastic/judge variation, not a new checkpoint. Printed precision retained, no recovery/delta recomputation. Quantized NVFP4 columns are separate unapproved reference variants and are not copied as BF16. Native mean output tokens per trajectory.

59.7% · raw 59.65 %

Headline · unknown effort · Own vendor

Source/record date: 2026-10-19

Harness: NVIDIA NeMo Gym/Evaluator, report Table22/§4.9. BF16 target, FP8 KV and FP32 Mamba SSM cache. Published saved-view result; card differences are explicitly attributed to stochastic/judge variation, not a new checkpoint. Printed precision retained, no recovery/delta recomputation. Quantized NVFP4 columns are separate unapproved reference variants and are not copied as BF16. Native published percent. CritPt zero is a genuine reported result; recovery is undefined, never zero. GDPval uses normalized committee Elo, not raw Elo.

7076 tokens/trajectory · raw 7076 tokens/trajectory

Headline · unknown effort · Own vendor

Source/record date: 2026-10-19

Harness: NVIDIA NeMo Gym/Evaluator, report Table22/§4.9. BF16 target, FP8 KV and FP32 Mamba SSM cache. Published saved-view result; card differences are explicitly attributed to stochastic/judge variation, not a new checkpoint. Printed precision retained, no recovery/delta recomputation. Quantized NVFP4 columns are separate unapproved reference variants and are not copied as BF16. Native mean output tokens per trajectory.

9.9% · raw 9.9 %

Headline · unknown effort · Own vendor

Source/record date: 2026-10-19

Harness: NVIDIA NeMo Gym/Evaluator, report Table22/§4.9. BF16 target, FP8 KV and FP32 Mamba SSM cache. Published saved-view result; card differences are explicitly attributed to stochastic/judge variation, not a new checkpoint. Printed precision retained, no recovery/delta recomputation. Quantized NVFP4 columns are separate unapproved reference variants and are not copied as BF16. Native published percent. CritPt zero is a genuine reported result; recovery is undefined, never zero. GDPval uses normalized committee Elo, not raw Elo.

19107 tokens/trajectory · raw 19107 tokens/trajectory

Headline · unknown effort · Own vendor

Source/record date: 2026-10-19

Harness: NVIDIA NeMo Gym/Evaluator, report Table22/§4.9. BF16 target, FP8 KV and FP32 Mamba SSM cache. Published saved-view result; card differences are explicitly attributed to stochastic/judge variation, not a new checkpoint. Printed precision retained, no recovery/delta recomputation. Quantized NVFP4 columns are separate unapproved reference variants and are not copied as BF16. Native mean output tokens per trajectory.

24.4% · raw 24.44 %

Headline · unknown effort · Own vendor

Source/record date: 2026-10-19

Harness: NVIDIA NeMo Gym/Evaluator, report Table22/§4.9. BF16 target, FP8 KV and FP32 Mamba SSM cache. Published saved-view result; card differences are explicitly attributed to stochastic/judge variation, not a new checkpoint. Printed precision retained, no recovery/delta recomputation. Quantized NVFP4 columns are separate unapproved reference variants and are not copied as BF16. Native published percent. CritPt zero is a genuine reported result; recovery is undefined, never zero. GDPval uses normalized committee Elo, not raw Elo.

76219 tokens/trajectory · raw 76219 tokens/trajectory

Headline · unknown effort · Own vendor

Source/record date: 2026-10-19

Harness: NVIDIA NeMo Gym/Evaluator, report Table22/§4.9. BF16 target, FP8 KV and FP32 Mamba SSM cache. Published saved-view result; card differences are explicitly attributed to stochastic/judge variation, not a new checkpoint. Printed precision retained, no recovery/delta recomputation. Quantized NVFP4 columns are separate unapproved reference variants and are not copied as BF16. Native mean output tokens per trajectory.

17.2% · raw 17.19 %

Headline · unknown effort · Own vendor

Source/record date: 2026-10-19

Harness: NVIDIA NeMo Gym/Evaluator, report Table22/§4.9. BF16 target, FP8 KV and FP32 Mamba SSM cache. Published saved-view result; card differences are explicitly attributed to stochastic/judge variation, not a new checkpoint. Printed precision retained, no recovery/delta recomputation. Quantized NVFP4 columns are separate unapproved reference variants and are not copied as BF16. Native published percent. CritPt zero is a genuine reported result; recovery is undefined, never zero. GDPval uses normalized committee Elo, not raw Elo.

52043 tokens/trajectory · raw 52043 tokens/trajectory

Headline · unknown effort · Own vendor

Source/record date: 2026-10-19

Harness: NVIDIA NeMo Gym/Evaluator, report Table22/§4.9. BF16 target, FP8 KV and FP32 Mamba SSM cache. Published saved-view result; card differences are explicitly attributed to stochastic/judge variation, not a new checkpoint. Printed precision retained, no recovery/delta recomputation. Quantized NVFP4 columns are separate unapproved reference variants and are not copied as BF16. Native mean output tokens per trajectory.

83.9% · raw 83.86 %

Headline · unknown effort · Own vendor

Source/record date: 2026-10-19

Harness: NVIDIA NeMo Gym/Evaluator, report Table22/§4.9. BF16 target, FP8 KV and FP32 Mamba SSM cache. Published saved-view result; card differences are explicitly attributed to stochastic/judge variation, not a new checkpoint. Printed precision retained, no recovery/delta recomputation. Quantized NVFP4 columns are separate unapproved reference variants and are not copied as BF16. Native published percent. CritPt zero is a genuine reported result; recovery is undefined, never zero. GDPval uses normalized committee Elo, not raw Elo.

7206 tokens/trajectory · raw 7206 tokens/trajectory

Headline · unknown effort · Own vendor

Source/record date: 2026-10-19

Harness: NVIDIA NeMo Gym/Evaluator, report Table22/§4.9. BF16 target, FP8 KV and FP32 Mamba SSM cache. Published saved-view result; card differences are explicitly attributed to stochastic/judge variation, not a new checkpoint. Printed precision retained, no recovery/delta recomputation. Quantized NVFP4 columns are separate unapproved reference variants and are not copied as BF16. Native mean output tokens per trajectory.

39.5% · raw 39.5 %

Headline · unknown effort · Own vendor

Source/record date: 2026-10-19

Harness: NVIDIA NeMo Gym/Evaluator, report Table22/§4.9. BF16 target, FP8 KV and FP32 Mamba SSM cache. Published saved-view result; card differences are explicitly attributed to stochastic/judge variation, not a new checkpoint. Printed precision retained, no recovery/delta recomputation. Quantized NVFP4 columns are separate unapproved reference variants and are not copied as BF16. Native published percent. CritPt zero is a genuine reported result; recovery is undefined, never zero. GDPval uses normalized committee Elo, not raw Elo.

142589 tokens/trajectory · raw 142589 tokens/trajectory

Headline · unknown effort · Own vendor

Source/record date: 2026-10-19

Harness: NVIDIA NeMo Gym/Evaluator, report Table22/§4.9. BF16 target, FP8 KV and FP32 Mamba SSM cache. Published saved-view result; card differences are explicitly attributed to stochastic/judge variation, not a new checkpoint. Printed precision retained, no recovery/delta recomputation. Quantized NVFP4 columns are separate unapproved reference variants and are not copied as BF16. Native mean output tokens per trajectory.

32.7% · raw 32.74 %

Headline · unknown effort · Own vendor

Source/record date: 2026-10-19

Harness: NVIDIA NeMo Gym/Evaluator, report Table22/§4.9. BF16 target, FP8 KV and FP32 Mamba SSM cache. Published saved-view result; card differences are explicitly attributed to stochastic/judge variation, not a new checkpoint. Printed precision retained, no recovery/delta recomputation. Quantized NVFP4 columns are separate unapproved reference variants and are not copied as BF16. Native published percent. CritPt zero is a genuine reported result; recovery is undefined, never zero. GDPval uses normalized committee Elo, not raw Elo.

17190 tokens/trajectory · raw 17190 tokens/trajectory

Headline · unknown effort · Own vendor

Source/record date: 2026-10-19

Harness: NVIDIA NeMo Gym/Evaluator, report Table22/§4.9. BF16 target, FP8 KV and FP32 Mamba SSM cache. Published saved-view result; card differences are explicitly attributed to stochastic/judge variation, not a new checkpoint. Printed precision retained, no recovery/delta recomputation. Quantized NVFP4 columns are separate unapproved reference variants and are not copied as BF16. Native mean output tokens per trajectory.

76.5% · raw 76.45 %

Headline · unknown effort · Own vendor

Source/record date: 2026-10-19

Harness: NVIDIA NeMo Gym/Evaluator, report Table22/§4.9. BF16 target, FP8 KV and FP32 Mamba SSM cache. Published saved-view result; card differences are explicitly attributed to stochastic/judge variation, not a new checkpoint. Printed precision retained, no recovery/delta recomputation. Quantized NVFP4 columns are separate unapproved reference variants and are not copied as BF16. Native published percent. CritPt zero is a genuine reported result; recovery is undefined, never zero. GDPval uses normalized committee Elo, not raw Elo.

15552 tokens/trajectory · raw 15552 tokens/trajectory

Headline · unknown effort · Own vendor

Source/record date: 2026-10-19

Harness: NVIDIA NeMo Gym/Evaluator, report Table22/§4.9. BF16 target, FP8 KV and FP32 Mamba SSM cache. Published saved-view result; card differences are explicitly attributed to stochastic/judge variation, not a new checkpoint. Printed precision retained, no recovery/delta recomputation. Quantized NVFP4 columns are separate unapproved reference variants and are not copied as BF16. Native mean output tokens per trajectory.

10.8% · raw 10.8 %

Headline · unknown effort · Own vendor

Source/record date: 2026-10-19

Harness: NVIDIA NeMo Gym/Evaluator, report Table22/§4.9. BF16 target, FP8 KV and FP32 Mamba SSM cache. Published saved-view result; card differences are explicitly attributed to stochastic/judge variation, not a new checkpoint. Printed precision retained, no recovery/delta recomputation. Quantized NVFP4 columns are separate unapproved reference variants and are not copied as BF16. Native published percent. CritPt zero is a genuine reported result; recovery is undefined, never zero. GDPval uses normalized committee Elo, not raw Elo.

30352 tokens/trajectory · raw 30352 tokens/trajectory

Headline · unknown effort · Own vendor

Source/record date: 2026-10-19

Harness: NVIDIA NeMo Gym/Evaluator, report Table22/§4.9. BF16 target, FP8 KV and FP32 Mamba SSM cache. Published saved-view result; card differences are explicitly attributed to stochastic/judge variation, not a new checkpoint. Printed precision retained, no recovery/delta recomputation. Quantized NVFP4 columns are separate unapproved reference variants and are not copied as BF16. Native mean output tokens per trajectory.

17.5% · raw 17.47 %

Headline · unknown effort · Own vendor

Source/record date: 2026-10-19

Harness: NVIDIA NeMo Gym/Evaluator, report Table22/§4.9. BF16 target, FP8 KV and FP32 Mamba SSM cache. Published saved-view result; card differences are explicitly attributed to stochastic/judge variation, not a new checkpoint. Printed precision retained, no recovery/delta recomputation. Quantized NVFP4 columns are separate unapproved reference variants and are not copied as BF16. Native published percent. CritPt zero is a genuine reported result; recovery is undefined, never zero. GDPval uses normalized committee Elo, not raw Elo.

2505 tokens/trajectory · raw 2505 tokens/trajectory

Headline · unknown effort · Own vendor

Source/record date: 2026-10-19

Harness: NVIDIA NeMo Gym/Evaluator, report Table22/§4.9. BF16 target, FP8 KV and FP32 Mamba SSM cache. Published saved-view result; card differences are explicitly attributed to stochastic/judge variation, not a new checkpoint. Printed precision retained, no recovery/delta recomputation. Quantized NVFP4 columns are separate unapproved reference variants and are not copied as BF16. Native mean output tokens per trajectory.

68.9% · raw 68.88 %

Headline · unknown effort · Own vendor

Source/record date: 2026-10-19

Harness: NVIDIA NeMo Gym/Evaluator, report Table22/§4.9. BF16 target, FP8 KV and FP32 Mamba SSM cache. Published saved-view result; card differences are explicitly attributed to stochastic/judge variation, not a new checkpoint. Printed precision retained, no recovery/delta recomputation. Quantized NVFP4 columns are separate unapproved reference variants and are not copied as BF16. Native published percent. CritPt zero is a genuine reported result; recovery is undefined, never zero. GDPval uses normalized committee Elo, not raw Elo.

2505 tokens/trajectory · raw 2505 tokens/trajectory

Headline · unknown effort · Own vendor

Source/record date: 2026-10-19

Harness: NVIDIA NeMo Gym/Evaluator, report Table22/§4.9. BF16 target, FP8 KV and FP32 Mamba SSM cache. Published saved-view result; card differences are explicitly attributed to stochastic/judge variation, not a new checkpoint. Printed precision retained, no recovery/delta recomputation. Quantized NVFP4 columns are separate unapproved reference variants and are not copied as BF16. Native mean output tokens per trajectory.

0% · raw 0 %

Headline · unknown effort · Own vendor

Source/record date: 2026-10-19

Harness: NVIDIA NeMo Gym/Evaluator, report Table22/§4.9. BF16 target, FP8 KV and FP32 Mamba SSM cache. Published saved-view result; card differences are explicitly attributed to stochastic/judge variation, not a new checkpoint. Printed precision retained, no recovery/delta recomputation. Quantized NVFP4 columns are separate unapproved reference variants and are not copied as BF16. Native published percent. CritPt zero is a genuine reported result; recovery is undefined, never zero. GDPval uses normalized committee Elo, not raw Elo.

4379 tokens/trajectory · raw 4379 tokens/trajectory

Headline · unknown effort · Own vendor

Source/record date: 2026-10-19

Harness: NVIDIA NeMo Gym/Evaluator, report Table22/§4.9. BF16 target, FP8 KV and FP32 Mamba SSM cache. Published saved-view result; card differences are explicitly attributed to stochastic/judge variation, not a new checkpoint. Printed precision retained, no recovery/delta recomputation. Quantized NVFP4 columns are separate unapproved reference variants and are not copied as BF16. Native mean output tokens per trajectory.

52.4% · raw 52.38 %

Headline · unknown effort · Own vendor

Source/record date: 2026-10-19

Harness: NVIDIA NeMo Gym/Evaluator, report Table22/§4.9. BF16 target, FP8 KV and FP32 Mamba SSM cache. Published saved-view result; card differences are explicitly attributed to stochastic/judge variation, not a new checkpoint. Printed precision retained, no recovery/delta recomputation. Quantized NVFP4 columns are separate unapproved reference variants and are not copied as BF16. Native published percent. CritPt zero is a genuine reported result; recovery is undefined, never zero. GDPval uses normalized committee Elo, not raw Elo.

7679 tokens/trajectory · raw 7679 tokens/trajectory

Headline · unknown effort · Own vendor

Source/record date: 2026-10-19

Harness: NVIDIA NeMo Gym/Evaluator, report Table22/§4.9. BF16 target, FP8 KV and FP32 Mamba SSM cache. Published saved-view result; card differences are explicitly attributed to stochastic/judge variation, not a new checkpoint. Printed precision retained, no recovery/delta recomputation. Quantized NVFP4 columns are separate unapproved reference variants and are not copied as BF16. Native mean output tokens per trajectory.

91.5% · raw 91.52 %

Headline · unknown effort · Own vendor

Source/record date: 2026-10-19

Harness: NVIDIA NeMo Gym/Evaluator, report Table22/§4.9. BF16 target, FP8 KV and FP32 Mamba SSM cache. Published saved-view result; card differences are explicitly attributed to stochastic/judge variation, not a new checkpoint. Printed precision retained, no recovery/delta recomputation. Quantized NVFP4 columns are separate unapproved reference variants and are not copied as BF16. Native published percent. CritPt zero is a genuine reported result; recovery is undefined, never zero. GDPval uses normalized committee Elo, not raw Elo.

22 tokens/trajectory · raw 22 tokens/trajectory

Headline · unknown effort · Own vendor

Source/record date: 2026-10-19

Harness: NVIDIA NeMo Gym/Evaluator, report Table22/§4.9. BF16 target, FP8 KV and FP32 Mamba SSM cache. Published saved-view result; card differences are explicitly attributed to stochastic/judge variation, not a new checkpoint. Printed precision retained, no recovery/delta recomputation. Quantized NVFP4 columns are separate unapproved reference variants and are not copied as BF16. Native mean output tokens per trajectory.

72% · raw 72 %

Headline · unknown effort · Own vendor

Source/record date: 2026-10-19

Harness: NVIDIA NeMo Gym/Evaluator, report Table22/§4.9. BF16 target, FP8 KV and FP32 Mamba SSM cache. Published saved-view result; card differences are explicitly attributed to stochastic/judge variation, not a new checkpoint. Printed precision retained, no recovery/delta recomputation. Quantized NVFP4 columns are separate unapproved reference variants and are not copied as BF16. Native published percent. CritPt zero is a genuine reported result; recovery is undefined, never zero. GDPval uses normalized committee Elo, not raw Elo.

13069 tokens/trajectory · raw 13069 tokens/trajectory

Headline · unknown effort · Own vendor

Source/record date: 2026-10-19

Harness: NVIDIA NeMo Gym/Evaluator, report Table22/§4.9. BF16 target, FP8 KV and FP32 Mamba SSM cache. Published saved-view result; card differences are explicitly attributed to stochastic/judge variation, not a new checkpoint. Printed precision retained, no recovery/delta recomputation. Quantized NVFP4 columns are separate unapproved reference variants and are not copied as BF16. Native mean output tokens per trajectory.

12.1% · raw 12.06 %

Headline · unknown effort · Own vendor

Source/record date: 2026-10-19

Harness: NVIDIA NeMo Gym/Evaluator, report Table22/§4.9. BF16 target, FP8 KV and FP32 Mamba SSM cache. Published saved-view result; card differences are explicitly attributed to stochastic/judge variation, not a new checkpoint. Printed precision retained, no recovery/delta recomputation. Quantized NVFP4 columns are separate unapproved reference variants and are not copied as BF16. Native published percent. CritPt zero is a genuine reported result; recovery is undefined, never zero. GDPval uses normalized committee Elo, not raw Elo.

3401 tokens/trajectory · raw 3401 tokens/trajectory

Headline · unknown effort · Own vendor

Source/record date: 2026-10-19

Harness: NVIDIA NeMo Gym/Evaluator, report Table22/§4.9. BF16 target, FP8 KV and FP32 Mamba SSM cache. Published saved-view result; card differences are explicitly attributed to stochastic/judge variation, not a new checkpoint. Printed precision retained, no recovery/delta recomputation. Quantized NVFP4 columns are separate unapproved reference variants and are not copied as BF16. Native mean output tokens per trajectory.

2.591% · raw 2.591 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Private193-item tax suite. All-pass accuracy and must-pass weighted partial credit are separate; both multiply by citation factor0.7+0.3*citation quality. No inferred tax professional reliability. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":1180.079,"stderr":1.146,"cost_per_test":0.030533,"token_totals":{"input_tokens":69555925,"output_tokens":22469385,"reasoning_tokens":14413630,"cache_read_tokens":51967923,"cache_write_tokens":null},"temperature":1.0,"top_p":0.95,"max_output_tokens":131072,"reasoning":null,"reasoning_effort":null,"verbosity":null,"compute_effort":null,"provider":"Fireworks AI","harness":null}

$0.030533/task · raw 0.030533 USD/task

Headline · unknown effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Private193-item tax suite. All-pass accuracy and must-pass weighted partial credit are separate; both multiply by citation factor0.7+0.3*citation quality. No inferred tax professional reliability. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":1180.079,"stderr":1.146,"cost_per_test":0.030533,"token_totals":{"input_tokens":69555925,"output_tokens":22469385,"reasoning_tokens":14413630,"cache_read_tokens":51967923,"cache_write_tokens":null},"temperature":1.0,"top_p":0.95,"max_output_tokens":131072,"reasoning":null,"reasoning_effort":null,"verbosity":null,"compute_effort":null,"provider":"Fireworks AI","harness":null} Source score standard error describes the benchmark quality score, not uncertainty in cost or latency.

1180.079 s · raw 1180.079 s

Headline · unknown effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Private193-item tax suite. All-pass accuracy and must-pass weighted partial credit are separate; both multiply by citation factor0.7+0.3*citation quality. No inferred tax professional reliability. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":1180.079,"stderr":1.146,"cost_per_test":0.030533,"token_totals":{"input_tokens":69555925,"output_tokens":22469385,"reasoning_tokens":14413630,"cache_read_tokens":51967923,"cache_write_tokens":null},"temperature":1.0,"top_p":0.95,"max_output_tokens":131072,"reasoning":null,"reasoning_effort":null,"verbosity":null,"compute_effort":null,"provider":"Fireworks AI","harness":null} Source score standard error describes the benchmark quality score, not uncertainty in cost or latency.

0.000% · raw 0 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Private193-item tax suite. All-pass accuracy and must-pass weighted partial credit are separate; both multiply by citation factor0.7+0.3*citation quality. No inferred tax professional reliability. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":1584.808,"stderr":0.0,"cost_per_test":0.034737,"temperature":1.0,"top_p":0.95,"max_output_tokens":131072,"reasoning":null,"reasoning_effort":null,"verbosity":null,"compute_effort":null,"provider":"Fireworks AI","harness":null}

10.000% · raw 10 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Private193-item tax suite. All-pass accuracy and must-pass weighted partial credit are separate; both multiply by citation factor0.7+0.3*citation quality. No inferred tax professional reliability. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":732.305,"stderr":5.571,"cost_per_test":0.020854,"temperature":1.0,"top_p":0.95,"max_output_tokens":131072,"reasoning":null,"reasoning_effort":null,"verbosity":null,"compute_effort":null,"provider":"Fireworks AI","harness":null}

3.333% · raw 3.333 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Private193-item tax suite. All-pass accuracy and must-pass weighted partial credit are separate; both multiply by citation factor0.7+0.3*citation quality. No inferred tax professional reliability. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":903.722,"stderr":3.333,"cost_per_test":0.027891,"temperature":1.0,"top_p":0.95,"max_output_tokens":131072,"reasoning":null,"reasoning_effort":null,"verbosity":null,"compute_effort":null,"provider":"Fireworks AI","harness":null}

3.333% · raw 3.333 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Private193-item tax suite. All-pass accuracy and must-pass weighted partial credit are separate; both multiply by citation factor0.7+0.3*citation quality. No inferred tax professional reliability. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":978.65,"stderr":3.333,"cost_per_test":0.024076,"temperature":1.0,"top_p":0.95,"max_output_tokens":131072,"reasoning":null,"reasoning_effort":null,"verbosity":null,"compute_effort":null,"provider":"Fireworks AI","harness":null}

0.000% · raw 0 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Private193-item tax suite. All-pass accuracy and must-pass weighted partial credit are separate; both multiply by citation factor0.7+0.3*citation quality. No inferred tax professional reliability. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":800.005,"stderr":0.0,"cost_per_test":0.028078,"temperature":1.0,"top_p":0.95,"max_output_tokens":131072,"reasoning":null,"reasoning_effort":null,"verbosity":null,"compute_effort":null,"provider":"Fireworks AI","harness":null}

0.000% · raw 0 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Private193-item tax suite. All-pass accuracy and must-pass weighted partial credit are separate; both multiply by citation factor0.7+0.3*citation quality. No inferred tax professional reliability. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":2038.781,"stderr":0.0,"cost_per_test":0.045743,"temperature":1.0,"top_p":0.95,"max_output_tokens":131072,"reasoning":null,"reasoning_effort":null,"verbosity":null,"compute_effort":null,"provider":"Fireworks AI","harness":null}

13.002% · raw 13.002 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Private193-item tax suite. All-pass accuracy and must-pass weighted partial credit are separate; both multiply by citation factor0.7+0.3*citation quality. No inferred tax professional reliability. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":1180.079,"stderr":2.049,"cost_per_test":0.030533,"token_totals":{"input_tokens":69555925,"output_tokens":22469385,"reasoning_tokens":14413630,"cache_read_tokens":51967923,"cache_write_tokens":null},"temperature":1.0,"top_p":0.95,"max_output_tokens":131072,"reasoning":null,"reasoning_effort":null,"verbosity":null,"compute_effort":null,"provider":"Fireworks AI","harness":null}

6.350% · raw 6.35 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Private193-item tax suite. All-pass accuracy and must-pass weighted partial credit are separate; both multiply by citation factor0.7+0.3*citation quality. No inferred tax professional reliability. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":1584.808,"stderr":2.727,"cost_per_test":0.034737,"temperature":1.0,"top_p":0.95,"max_output_tokens":131072,"reasoning":null,"reasoning_effort":null,"verbosity":null,"compute_effort":null,"provider":"Fireworks AI","harness":null}

14.685% · raw 14.685 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Private193-item tax suite. All-pass accuracy and must-pass weighted partial credit are separate; both multiply by citation factor0.7+0.3*citation quality. No inferred tax professional reliability. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":732.305,"stderr":6.205,"cost_per_test":0.020854,"temperature":1.0,"top_p":0.95,"max_output_tokens":131072,"reasoning":null,"reasoning_effort":null,"verbosity":null,"compute_effort":null,"provider":"Fireworks AI","harness":null}

10.469% · raw 10.469 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Private193-item tax suite. All-pass accuracy and must-pass weighted partial credit are separate; both multiply by citation factor0.7+0.3*citation quality. No inferred tax professional reliability. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":903.722,"stderr":5.136,"cost_per_test":0.027891,"temperature":1.0,"top_p":0.95,"max_output_tokens":131072,"reasoning":null,"reasoning_effort":null,"verbosity":null,"compute_effort":null,"provider":"Fireworks AI","harness":null}

31.499% · raw 31.499 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Private193-item tax suite. All-pass accuracy and must-pass weighted partial credit are separate; both multiply by citation factor0.7+0.3*citation quality. No inferred tax professional reliability. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":978.65,"stderr":6.731,"cost_per_test":0.024076,"temperature":1.0,"top_p":0.95,"max_output_tokens":131072,"reasoning":null,"reasoning_effort":null,"verbosity":null,"compute_effort":null,"provider":"Fireworks AI","harness":null}

6.661% · raw 6.661 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Private193-item tax suite. All-pass accuracy and must-pass weighted partial credit are separate; both multiply by citation factor0.7+0.3*citation quality. No inferred tax professional reliability. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":800.005,"stderr":3.751,"cost_per_test":0.028078,"temperature":1.0,"top_p":0.95,"max_output_tokens":131072,"reasoning":null,"reasoning_effort":null,"verbosity":null,"compute_effort":null,"provider":"Fireworks AI","harness":null}

11.232% · raw 11.232 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-10-20

Harness: publisher benchmark protocol. Private193-item tax suite. All-pass accuracy and must-pass weighted partial credit are separate; both multiply by citation factor0.7+0.3*citation quality. No inferred tax professional reliability. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":2038.781,"stderr":4.839,"cost_per_test":0.045743,"temperature":1.0,"top_p":0.95,"max_output_tokens":131072,"reasoning":null,"reasoning_effort":null,"verbosity":null,"compute_effort":null,"provider":"Fireworks AI","harness":null}

0.000% · raw 0 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-10-20

Harness: OpenHands. Independent model-plus-harness web-application evaluation. Preserve OpenHands versus other agent variants; source costs/runtime are task observations, not API tariffs. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":959.662,"stderr":0.0,"cost_per_test":0.128799,"token_totals":{"input_tokens":535193452,"output_tokens":3346611,"reasoning_tokens":null,"cache_read_tokens":null,"cache_write_tokens":null},"temperature":1.0,"top_p":0.95,"max_output_tokens":131072,"reasoning":null,"reasoning_effort":null,"verbosity":null,"compute_effort":null,"provider":"Fireworks AI","harness":"OpenHands"}

0.000% · raw 0 %

Alternative · unknown effort · Independent evaluator

Source/record date: 2026-10-19

Independent Vals current model-page card; Vibe Code Bench v1.1. Published uncertainty±0.00; rank111/113 at source check. Default NVIDIA,temperature1,top_p0.95,max output131072; effort not stated. Individual benchmarks may override defaults. Demoted 2026-10-20: current publisher observation from https://www.vals.ai/benchmarks/vibe-code is the headline for this same version/configuration family; this historical value and source remain available. Revised benchmark versions use separate metrics.

$0.128799/task · raw 0.128799 USD/task

Headline · unknown effort · Independent evaluator

Source/record date: 2026-10-20

Harness: OpenHands. Independent model-plus-harness web-application evaluation. Preserve OpenHands versus other agent variants; source costs/runtime are task observations, not API tariffs. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":959.662,"stderr":0.0,"cost_per_test":0.128799,"token_totals":{"input_tokens":535193452,"output_tokens":3346611,"reasoning_tokens":null,"cache_read_tokens":null,"cache_write_tokens":null},"temperature":1.0,"top_p":0.95,"max_output_tokens":131072,"reasoning":null,"reasoning_effort":null,"verbosity":null,"compute_effort":null,"provider":"Fireworks AI","harness":"OpenHands"} Source score standard error describes the benchmark quality score, not uncertainty in cost or latency.

959.662 s · raw 959.662 s

Headline · unknown effort · Independent evaluator

Source/record date: 2026-10-20

Harness: OpenHands. Independent model-plus-harness web-application evaluation. Preserve OpenHands versus other agent variants; source costs/runtime are task observations, not API tariffs. Published summary updated 19 October 2026; exact run dates and serving weight SHA unpublished. Requested effort from published reasoning_effort/compute_effort is not a common compute budget. Raw source diagnostics: {"latency":959.662,"stderr":0.0,"cost_per_test":0.128799,"token_totals":{"input_tokens":535193452,"output_tokens":3346611,"reasoning_tokens":null,"cache_read_tokens":null,"cache_write_tokens":null},"temperature":1.0,"top_p":0.95,"max_output_tokens":131072,"reasoning":null,"reasoning_effort":null,"verbosity":null,"compute_effort":null,"provider":"Fireworks AI","harness":"OpenHands"} Source score standard error describes the benchmark quality score, not uncertainty in cost or latency.

639 · raw 639 Elo

Headline · unknown effort · Independent evaluator

Source/record date: 2026-10-20

Manually observed public GDPval-AA v2.1 leaderboard table, full 298-configuration pool checked 20 October 2026. Exact source configuration: Nemotron 3.5 Lightning; creator: NVIDIA; source rank 209; source release-month label Aug 2026 is model release metadata, not execution date. Published rounded native Elo 639; literal displayed CI -24 / +24. Native Elo is not a percentage and is not the normalized (Elo−500)/2000 chart quantity. This versioned metric is separate from older generic GDPval-AA Elo and vendor GDPval/GDPval-AA results; no old evidence is overwritten. Execution date, inference precision, unprinted effort budgets and task costs are unpublished in this captured table. Elo is relative to the dated comparison pool; drift from an earlier observation does not prove a new evaluation run. Source does not label an effort; unknown is preserved rather than copying a vendor default. AA exact model-page links to the approved reference checkpoint were verified in prior curated evidence; a BF16 reference link does not prove BF16 inference. Headline: highest published native Elo for this exact approved model; tied values prefer existing featured effort then source order.

12.9 · raw 12.8572306217144 index

Headline · unknown effort · Independent evaluator

Source/record date: 2026-10-19

Independent AA exact model page explicitly links https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16. Reasoning enabled without a published common API effort label. Index v4.3.2; inference precision, execution date and exact budget unpublished. Linking reference weights does not prove BF16 inference. Costs with unavailable/zero model prices are not imported as free tariffs.

-17.7 · raw -17.716666666666665 index

Headline · unknown effort · Independent evaluator

Source/record date: 2026-10-19

Independent AA exact model page explicitly links https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16. Reasoning enabled without a published common API effort label. Index v4.3.2; inference precision, execution date and exact budget unpublished. Linking reference weights does not prove BF16 inference. Costs with unavailable/zero model prices are not imported as free tariffs.

14.4% · raw 14.433333333333334 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-10-19

Independent AA exact model page explicitly links https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16. Reasoning enabled without a published common API effort label. Index v4.3.2; inference precision, execution date and exact budget unpublished. Linking reference weights does not prove BF16 inference. Costs with unavailable/zero model prices are not imported as free tariffs.

37.6% · raw 37.573042462017916 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-10-19

Independent AA exact model page explicitly links https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16. Reasoning enabled without a published common API effort label. Index v4.3.2; inference precision, execution date and exact budget unpublished. Linking reference weights does not prove BF16 inference. Costs with unavailable/zero model prices are not imported as free tariffs.

638.6 · raw 638.57 Elo

Headline · unknown effort · Independent evaluator

Source/record date: 2026-10-19

Independent AA exact model page explicitly links https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16. Reasoning enabled without a published common API effort label. Index v4.3.2; inference precision, execution date and exact budget unpublished. Linking reference weights does not prove BF16 inference. Costs with unavailable/zero model prices are not imported as free tariffs.

0.5% · raw 0.505050505050505 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-10-19

Independent AA exact model page explicitly links https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16. Reasoning enabled without a published common API effort label. Index v4.3.2; inference precision, execution date and exact budget unpublished. Linking reference weights does not prove BF16 inference. Costs with unavailable/zero model prices are not imported as free tariffs.

24.3% · raw 24.3445692883895 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-10-19

Independent AA exact model page explicitly links https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16. Reasoning enabled without a published common API effort label. Index v4.3.2; inference precision, execution date and exact budget unpublished. Linking reference weights does not prove BF16 inference. Costs with unavailable/zero model prices are not imported as free tariffs.

32.1% · raw 32.060185185185205 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-10-19

Independent AA exact model page explicitly links https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16. Reasoning enabled without a published common API effort label. Index v4.3.2; inference precision, execution date and exact budget unpublished. Linking reference weights does not prove BF16 inference. Costs with unavailable/zero model prices are not imported as free tariffs.

60.3% · raw 60.33333333333331 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-10-19

Independent AA exact model page explicitly links https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16. Reasoning enabled without a published common API effort label. Index v4.3.2; inference precision, execution date and exact budget unpublished. Linking reference weights does not prove BF16 inference. Costs with unavailable/zero model prices are not imported as free tariffs.

10.6% · raw 10.5653382761817 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-10-19

Independent AA exact model page explicitly links https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16. Reasoning enabled without a published common API effort label. Index v4.3.2; inference precision, execution date and exact budget unpublished. Linking reference weights does not prove BF16 inference. Costs with unavailable/zero model prices are not imported as free tariffs.

74.3% · raw 74.3434343434343 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-10-19

Independent AA exact model page explicitly links https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16. Reasoning enabled without a published common API effort label. Index v4.3.2; inference precision, execution date and exact budget unpublished. Linking reference weights does not prove BF16 inference. Costs with unavailable/zero model prices are not imported as free tariffs.

0% · raw 0 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-10-19

Independent AA exact model page explicitly links https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16. Reasoning enabled without a published common API effort label. Index v4.3.2; inference precision, execution date and exact budget unpublished. Linking reference weights does not prove BF16 inference. Costs with unavailable/zero model prices are not imported as free tariffs.

3% · raw 3 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-10-19

Independent AA exact model page explicitly links https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16. Reasoning enabled without a published common API effort label. Index v4.3.2; inference precision, execution date and exact budget unpublished. Linking reference weights does not prove BF16 inference. Costs with unavailable/zero model prices are not imported as free tariffs.

0.8% · raw 0.8410069634719541 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-10-19

Independent AA exact model page explicitly links https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16. Reasoning enabled without a published common API effort label. Index v4.3.2; inference precision, execution date and exact budget unpublished. Linking reference weights does not prove BF16 inference. Costs with unavailable/zero model prices are not imported as free tariffs.

8.9% · raw 8.8659793814433 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-10-19

Independent AA exact model page explicitly links https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16. Reasoning enabled without a published common API effort label. Index v4.3.2; inference precision, execution date and exact budget unpublished. Linking reference weights does not prove BF16 inference. Costs with unavailable/zero model prices are not imported as free tariffs.

18.4% · raw 18.352730528200535 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-10-19

Independent AA exact model page explicitly links https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16. Reasoning enabled without a published common API effort label. Index v4.3.2; inference precision, execution date and exact budget unpublished. Linking reference weights does not prove BF16 inference. Costs with unavailable/zero model prices are not imported as free tariffs.

12.2 · raw 12.2355771675463 index

Headline · unknown effort · Independent evaluator

Source/record date: 2026-10-19

Independent AA exact model page explicitly links https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16. Reasoning enabled without a published common API effort label. Index v4.3.2; inference precision, execution date and exact budget unpublished. Linking reference weights does not prove BF16 inference. Costs with unavailable/zero model prices are not imported as free tariffs.

6.9 · raw 6.93936490339053 index

Headline · unknown effort · Independent evaluator

Source/record date: 2026-10-19

Independent AA exact model page explicitly links https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16. Reasoning enabled without a published common API effort label. Index v4.3.2; inference precision, execution date and exact budget unpublished. Linking reference weights does not prove BF16 inference. Costs with unavailable/zero model prices are not imported as free tariffs.

15.5 · raw 15.4761748762967 index

Headline · unknown effort · Independent evaluator

Source/record date: 2026-10-19

Independent AA exact model page explicitly links https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16. Reasoning enabled without a published common API effort label. Index v4.3.2; inference precision, execution date and exact budget unpublished. Linking reference weights does not prove BF16 inference. Costs with unavailable/zero model prices are not imported as free tariffs.

18.1 · raw 18.0652083171848 index

Headline · unknown effort · Independent evaluator

Source/record date: 2026-10-19

Independent AA exact model page explicitly links https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16. Reasoning enabled without a published common API effort label. Index v4.3.2; inference precision, execution date and exact budget unpublished. Linking reference weights does not prove BF16 inference. Costs with unavailable/zero model prices are not imported as free tariffs.

22.8 · raw 22.8015975633303 index

Headline · unknown effort · Independent evaluator

Source/record date: 2026-10-19

Independent AA exact model page explicitly links https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16. Reasoning enabled without a published common API effort label. Index v4.3.2; inference precision, execution date and exact budget unpublished. Linking reference weights does not prove BF16 inference. Costs with unavailable/zero model prices are not imported as free tariffs.

506.2 · raw 506.17 Elo

Headline · unknown effort · Independent evaluator

Source/record date: 2026-10-19

Independent AA exact model page explicitly links https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16. Reasoning enabled without a published common API effort label. Index v4.3.2; inference precision, execution date and exact budget unpublished. Linking reference weights does not prove BF16 inference. Costs with unavailable/zero model prices are not imported as free tariffs.

351.5 · raw 351.5 Elo

Headline · unknown effort · Independent evaluator

Source/record date: 2026-10-19

Independent AA exact model page explicitly links https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16. Reasoning enabled without a published common API effort label. Index v4.3.2; inference precision, execution date and exact budget unpublished. Linking reference weights does not prove BF16 inference. Costs with unavailable/zero model prices are not imported as free tariffs.

491.5 · raw 491.49 Elo

Headline · unknown effort · Independent evaluator

Source/record date: 2026-10-19

Independent AA exact model page explicitly links https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16. Reasoning enabled without a published common API effort label. Index v4.3.2; inference precision, execution date and exact budget unpublished. Linking reference weights does not prove BF16 inference. Costs with unavailable/zero model prices are not imported as free tariffs.

8.9% · raw 8.88888888888889 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-10-19

Independent AA exact model page explicitly links https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16. Reasoning enabled without a published common API effort label. Index v4.3.2; inference precision, execution date and exact budget unpublished. Linking reference weights does not prove BF16 inference. Costs with unavailable/zero model prices are not imported as free tariffs.

298.5 tok/s · raw 298.537053079632 tok/s

Headline · unknown effort · Independent evaluator

Source/record date: 2026-10-19

Independent AA exact model page explicitly links https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16. Reasoning enabled without a published common API effort label. Index v4.3.2; inference precision, execution date and exact budget unpublished. Linking reference weights does not prove BF16 inference. Costs with unavailable/zero model prices are not imported as free tariffs.

3% · raw 2.95 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-10-19

Independent Vals current model-page card; Code Migration. Published uncertainty±0.95; rank75/78 at source check. Default NVIDIA,temperature1,top_p0.95,max output131072; effort not stated. Individual benchmarks may override defaults.

29.5% · raw 29.46 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-10-19

Independent Vals current model-page card; CyberBench v1.1. Published uncertainty±5.96; rank46/47 at source check. Default NVIDIA,temperature1,top_p0.95,max output131072; effort not stated. Individual benchmarks may override defaults.

17% · raw 16.99 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-10-19

Independent Vals current model-page card; EMB. Published uncertainty±1.45; rank71/75 at source check. Default NVIDIA,temperature1,top_p0.95,max output131072; effort not stated. Individual benchmarks may override defaults.

18.5% · raw 18.5 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-10-19

Independent Vals current model-page card; Finance Agent (v2). Published uncertainty±0.88; rank77/79 at source check. Default NVIDIA,temperature1,top_p0.95,max output131072; effort not stated. Individual benchmarks may override defaults.

2.9% · raw 2.88 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-10-19

Independent Vals current model-page card; Legal Research Bench. Published uncertainty±1.16; rank76/79 at source check. Default NVIDIA,temperature1,top_p0.95,max output131072; effort not stated. Individual benchmarks may override defaults.

4.3% · raw 4.27 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-10-19

Independent Vals current model-page card; MedScribe. Published uncertainty±0.49; rank109/109 at source check. Default NVIDIA,temperature1,top_p0.95,max output131072; effort not stated. Individual benchmarks may override defaults.

0% · raw 0 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-10-19

Independent Vals current model-page card; ProofBench v1.1. Published uncertainty±0.00; rank50/50 at source check. Default NVIDIA,temperature1,top_p0.95,max output131072; effort not stated. Individual benchmarks may override defaults.

2.6% · raw 2.59 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-10-19

Independent Vals current model-page card; Tax Agent Bench. Published uncertainty±1.15; rank69/70 at source check. Default NVIDIA,temperature1,top_p0.95,max output131072; effort not stated. Individual benchmarks may override defaults. Legacy mixed all-pass/weighted quantity; not a common ranking.

0% · raw 0 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-10-19

Independent Vals current model-page card; Harvey's Legal Agent Benchmark. Published uncertainty±0.00; rank80/80 at source check. Default NVIDIA,temperature1,top_p0.95,max output131072; effort not stated. Individual benchmarks may override defaults.

0% · raw 0 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-10-19

Independent Vals current model-page card; ProgramBench. Published uncertainty±0.00; rank59/64 at source check. Default NVIDIA,temperature1,top_p0.95,max output131072; effort not stated. Individual benchmarks may override defaults.

52.8% · raw 52.8 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-10-19

Independent Vals current model-page card; SWE-bench. Published uncertainty±2.23; rank83/88 at source check. Default NVIDIA,temperature1,top_p0.95,max output131072; effort not stated. Individual benchmarks may override defaults.

10.9% · raw 10.86 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-10-19

Independent Vals current model-page card; Terminal-Bench 2.1. Published uncertainty±2.62; rank75/75 at source check. Default NVIDIA,temperature1,top_p0.95,max output131072; effort not stated. Individual benchmarks may override defaults.

10.870% · raw 10.87 %

Headline · unknown effort · Independent evaluator

Source/record date: 2026-10-19

Current Vals model-page published aggregate; vendor metadata and token prices are not copied from this evaluator. Default NVIDIA,temperature1,top_p0.95,max output131072; exact effort/run date unknown.