Nimble-v4-max Decision Model Results

Nimble-v4-max

Explore Nimble-v4-max: exact decision configuration, published specifications, source-linked benchmarks and API tariffs when available.

Published specifications

Provider: Bespoke Labs
Version: bespokelabs/Nimble-v4-max
System type: Trained checkpoint
Access: Open weights
License: CC BY-NC 4.0
Inputs: text
Decision types: Choice, Yes / no, Score
Published size: 27.36B
Active parameters: Not published
Context / input limit: Not published
Announced: 2026-10-13

Hugging Face · Announcement · Website

Model notes

Fine-tune of Qwen/Qwen3.8-27B; about 55 GB in BF16, so it needs one 80 GB or larger GPU. The hosted API launched 13 October 2026 and the weights followed on 18 October. nimble-latest is a moving alias and nimble-v4 is a deprecated alias, so neither is a separate entry. Nimble-v4 (9B) and the older Bespoke-Nimble-9B releases are not registered. Typed outputs are probabilities over supplied options; confidence is not a correctness guarantee.

Checkpoint revision: 655aa7051c2ab39660193ebb70c8926d3d2cb7aa

← All decision models · Compare this decision system →

Published API tariff

$0.04 per 1M input tokens

Bespoke Labs · checked 2026-11-02

Bespoke pricing page: $0.04 per 1M input tokens. Output tokens are not charged on a decision model.

No output-token charge does not mean free requests. These are provider tariffs, separate from benchmark workload costs.

377 recorded observations. Published results retain their source and setup; no Examenos evaluation was run. Missing results are not zero. How to read these comparisons →

Official / vendor · Nimble-v4-max model card (vendor measured)

Nimble-v4-max · Official / vendor · Nimble-v4-max model card (vendor measured)
Evaluation	Result	Evidence
Nimble-v4-max card · Decision Index 0.3 public suite	68.46	Setup & source Own run of the public 0.3 suite. Own run; results in bespokelabs/nimble-v4-max-decision-index. Printed in the vendor's own table. Bespoke Labs' own run of the public 0.3 suite on 12 October 2026, published with a full results dataset. Not ranked here. Checked / reported: 2026-11-02 · Samples: Not published Snapshot: 2026-10-18 · Revision: 655aa7051c2ab39660193ebb70c8926d3d2cb7aa · Published observations; result reuse terms not specified Result source · Benchmark source

Evaluator-published · Community Decision Index

Evaluator-published · etsabary System 1 Decision Benchmark

Nimble-v4-max · Evaluator-published · etsabary System 1 Decision Benchmark
Evaluation	Result	Evidence
System 1 Decision Benchmark v2 · All-question accuracy	79.886%	Setup & source Correct answers over all 1,934 questions. Nimble v4 Max; Hosted native; returned identities {"nimble-v4-max": 1933, "unreported in selected primary": 1}; collected 2026-10-16 to 2026-10-31 America/Toronto; one primary response per input, no scored retries. Unsuccessful responses counted as wrong: {"other provider failure": 1}. Development benchmark, not an independently sampled test: 1,934 questions in 41 categories, the frozen 1,342-question bank plus 192 logical and 400 practical additions. The bank was built with model-guided development, and Jev plus retained Microsoft and Drex feedback are development exposure. Related variants and a single recorded primary response per input limit generalization. The scoring key stays private, so public checks cover coverage, timing and aggregate consistency but cannot rescore answers. Strict accuracy counts a wrong usable answer and an unsuccessful response in the same fixed denominator. Timing pools primary client times with the network included; providers, precision, concurrency, routes and dates differ, retained and new timings can span sessions, and quick failures can reduce all-primary latency. The source published a separate cost ledger on 1 November 2026, stored as decision-reference-v2-cost-per-1000, which supersedes its earlier “cost not consolidated” note. Older 1,342-only scores are not converted into this bank. Not ranked here. Checked / reported: 2026-11-02 · Samples: 1934 Snapshot: 2026-11-01 · Revision: f5fa9cfafccd97fcfbdb0336e14284faaa8d2d35 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
System 1 Decision Benchmark v2 · Macro-category accuracy	79.845%	Setup & source Equal-weight mean over the 41 categories. Nimble v4 Max; Hosted native; returned identities {"nimble-v4-max": 1933, "unreported in selected primary": 1}; collected 2026-10-16 to 2026-10-31 America/Toronto; one primary response per input, no scored retries. Unsuccessful responses counted as wrong: {"other provider failure": 1}. Development benchmark, not an independently sampled test: 1,934 questions in 41 categories, the frozen 1,342-question bank plus 192 logical and 400 practical additions. The bank was built with model-guided development, and Jev plus retained Microsoft and Drex feedback are development exposure. Related variants and a single recorded primary response per input limit generalization. The scoring key stays private, so public checks cover coverage, timing and aggregate consistency but cannot rescore answers. Strict accuracy counts a wrong usable answer and an unsuccessful response in the same fixed denominator. Timing pools primary client times with the network included; providers, precision, concurrency, routes and dates differ, retained and new timings can span sessions, and quick failures can reduce all-primary latency. The source published a separate cost ledger on 1 November 2026, stored as decision-reference-v2-cost-per-1000, which supersedes its earlier “cost not consolidated” note. Older 1,342-only scores are not converted into this bank. Not ranked here. Checked / reported: 2026-11-02 · Samples: 1934 Snapshot: 2026-11-01 · Revision: f5fa9cfafccd97fcfbdb0336e14284faaa8d2d35 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
System 1 Decision Benchmark v2 · Added 592 questions	72.635%	Setup & source Accuracy on the 592 questions new in v2. Nimble v4 Max; Hosted native; returned identities {"nimble-v4-max": 1933, "unreported in selected primary": 1}; collected 2026-10-16 to 2026-10-31 America/Toronto; one primary response per input, no scored retries. Unsuccessful responses counted as wrong: {"other provider failure": 1}. Development benchmark, not an independently sampled test: 1,934 questions in 41 categories, the frozen 1,342-question bank plus 192 logical and 400 practical additions. The bank was built with model-guided development, and Jev plus retained Microsoft and Drex feedback are development exposure. Related variants and a single recorded primary response per input limit generalization. The scoring key stays private, so public checks cover coverage, timing and aggregate consistency but cannot rescore answers. Strict accuracy counts a wrong usable answer and an unsuccessful response in the same fixed denominator. Timing pools primary client times with the network included; providers, precision, concurrency, routes and dates differ, retained and new timings can span sessions, and quick failures can reduce all-primary latency. The source published a separate cost ledger on 1 November 2026, stored as decision-reference-v2-cost-per-1000, which supersedes its earlier “cost not consolidated” note. Older 1,342-only scores are not converted into this bank. Not ranked here. Checked / reported: 2026-11-02 · Samples: 592 Snapshot: 2026-11-01 · Revision: f5fa9cfafccd97fcfbdb0336e14284faaa8d2d35 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
System 1 Decision Benchmark v2 · Logical additions	65.104%	Setup & source Accuracy on the 192 logical additions. Nimble v4 Max; Hosted native; returned identities {"nimble-v4-max": 1933, "unreported in selected primary": 1}; collected 2026-10-16 to 2026-10-31 America/Toronto; one primary response per input, no scored retries. Unsuccessful responses counted as wrong: {"other provider failure": 1}. Development benchmark, not an independently sampled test: 1,934 questions in 41 categories, the frozen 1,342-question bank plus 192 logical and 400 practical additions. The bank was built with model-guided development, and Jev plus retained Microsoft and Drex feedback are development exposure. Related variants and a single recorded primary response per input limit generalization. The scoring key stays private, so public checks cover coverage, timing and aggregate consistency but cannot rescore answers. Strict accuracy counts a wrong usable answer and an unsuccessful response in the same fixed denominator. Timing pools primary client times with the network included; providers, precision, concurrency, routes and dates differ, retained and new timings can span sessions, and quick failures can reduce all-primary latency. The source published a separate cost ledger on 1 November 2026, stored as decision-reference-v2-cost-per-1000, which supersedes its earlier “cost not consolidated” note. Older 1,342-only scores are not converted into this bank. Not ranked here. Checked / reported: 2026-11-02 · Samples: 192 Snapshot: 2026-11-01 · Revision: f5fa9cfafccd97fcfbdb0336e14284faaa8d2d35 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
System 1 Decision Benchmark v2 · Practical additions	76.250%	Setup & source Accuracy on the 400 practical additions. Nimble v4 Max; Hosted native; returned identities {"nimble-v4-max": 1933, "unreported in selected primary": 1}; collected 2026-10-16 to 2026-10-31 America/Toronto; one primary response per input, no scored retries. Unsuccessful responses counted as wrong: {"other provider failure": 1}. Development benchmark, not an independently sampled test: 1,934 questions in 41 categories, the frozen 1,342-question bank plus 192 logical and 400 practical additions. The bank was built with model-guided development, and Jev plus retained Microsoft and Drex feedback are development exposure. Related variants and a single recorded primary response per input limit generalization. The scoring key stays private, so public checks cover coverage, timing and aggregate consistency but cannot rescore answers. Strict accuracy counts a wrong usable answer and an unsuccessful response in the same fixed denominator. Timing pools primary client times with the network included; providers, precision, concurrency, routes and dates differ, retained and new timings can span sessions, and quick failures can reduce all-primary latency. The source published a separate cost ledger on 1 November 2026, stored as decision-reference-v2-cost-per-1000, which supersedes its earlier “cost not consolidated” note. Older 1,342-only scores are not converted into this bank. Not ranked here. Checked / reported: 2026-11-02 · Samples: 400 Snapshot: 2026-11-01 · Revision: f5fa9cfafccd97fcfbdb0336e14284faaa8d2d35 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
System 1 Decision Benchmark v2 · Original 1,342 questions	83.085%	Setup & source Accuracy on the frozen original inputs inside the v2 bank. Nimble v4 Max; Hosted native; returned identities {"nimble-v4-max": 1933, "unreported in selected primary": 1}; collected 2026-10-16 to 2026-10-31 America/Toronto; one primary response per input, no scored retries. Unsuccessful responses counted as wrong: {"other provider failure": 1}. Development benchmark, not an independently sampled test: 1,934 questions in 41 categories, the frozen 1,342-question bank plus 192 logical and 400 practical additions. The bank was built with model-guided development, and Jev plus retained Microsoft and Drex feedback are development exposure. Related variants and a single recorded primary response per input limit generalization. The scoring key stays private, so public checks cover coverage, timing and aggregate consistency but cannot rescore answers. Strict accuracy counts a wrong usable answer and an unsuccessful response in the same fixed denominator. Timing pools primary client times with the network included; providers, precision, concurrency, routes and dates differ, retained and new timings can span sessions, and quick failures can reduce all-primary latency. The source published a separate cost ledger on 1 November 2026, stored as decision-reference-v2-cost-per-1000, which supersedes its earlier “cost not consolidated” note. Older 1,342-only scores are not converted into this bank. Not ranked here. Checked / reported: 2026-11-02 · Samples: 1342 Snapshot: 2026-11-01 · Revision: f5fa9cfafccd97fcfbdb0336e14284faaa8d2d35 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
System 1 Decision Benchmark v2 · Median client time	238.8 ms	Setup & source Pooled primary complete-response client time, network included. Nimble v4 Max; Hosted native; returned identities {"nimble-v4-max": 1933, "unreported in selected primary": 1}; collected 2026-10-16 to 2026-10-31 America/Toronto; one primary response per input, no scored retries. Unsuccessful responses counted as wrong: {"other provider failure": 1}. Development benchmark, not an independently sampled test: 1,934 questions in 41 categories, the frozen 1,342-question bank plus 192 logical and 400 practical additions. The bank was built with model-guided development, and Jev plus retained Microsoft and Drex feedback are development exposure. Related variants and a single recorded primary response per input limit generalization. The scoring key stays private, so public checks cover coverage, timing and aggregate consistency but cannot rescore answers. Strict accuracy counts a wrong usable answer and an unsuccessful response in the same fixed denominator. Timing pools primary client times with the network included; providers, precision, concurrency, routes and dates differ, retained and new timings can span sessions, and quick failures can reduce all-primary latency. The source published a separate cost ledger on 1 November 2026, stored as decision-reference-v2-cost-per-1000, which supersedes its earlier “cost not consolidated” note. Older 1,342-only scores are not converted into this bank. Not ranked here. Checked / reported: 2026-11-02 · Samples: 1934 Snapshot: 2026-11-01 · Revision: f5fa9cfafccd97fcfbdb0336e14284faaa8d2d35 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
System 1 Decision Benchmark v2 · P95 client time	325.9 ms	Setup & source Pooled primary complete-response client time, network included. Nimble v4 Max; Hosted native; returned identities {"nimble-v4-max": 1933, "unreported in selected primary": 1}; collected 2026-10-16 to 2026-10-31 America/Toronto; one primary response per input, no scored retries. Unsuccessful responses counted as wrong: {"other provider failure": 1}. Development benchmark, not an independently sampled test: 1,934 questions in 41 categories, the frozen 1,342-question bank plus 192 logical and 400 practical additions. The bank was built with model-guided development, and Jev plus retained Microsoft and Drex feedback are development exposure. Related variants and a single recorded primary response per input limit generalization. The scoring key stays private, so public checks cover coverage, timing and aggregate consistency but cannot rescore answers. Strict accuracy counts a wrong usable answer and an unsuccessful response in the same fixed denominator. Timing pools primary client times with the network included; providers, precision, concurrency, routes and dates differ, retained and new timings can span sessions, and quick failures can reduce all-primary latency. The source published a separate cost ledger on 1 November 2026, stored as decision-reference-v2-cost-per-1000, which supersedes its earlier “cost not consolidated” note. Older 1,342-only scores are not converted into this bank. Not ranked here. Checked / reported: 2026-11-02 · Samples: 1934 Snapshot: 2026-11-01 · Revision: f5fa9cfafccd97fcfbdb0336e14284faaa8d2d35 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
System 1 Decision Benchmark v2 · F01 · Lookup and filtering	95.349%	Setup & source Lookup and filtering: 43 questions. Nimble v4 Max; Hosted native; returned identities {"nimble-v4-max": 1933, "unreported in selected primary": 1}; collected 2026-10-16 to 2026-10-31 America/Toronto; one primary response per input, no scored retries. Unsuccessful responses counted as wrong: {"other provider failure": 1}. Development benchmark, not an independently sampled test: 1,934 questions in 41 categories, the frozen 1,342-question bank plus 192 logical and 400 practical additions. The bank was built with model-guided development, and Jev plus retained Microsoft and Drex feedback are development exposure. Related variants and a single recorded primary response per input limit generalization. The scoring key stays private, so public checks cover coverage, timing and aggregate consistency but cannot rescore answers. Strict accuracy counts a wrong usable answer and an unsuccessful response in the same fixed denominator. Timing pools primary client times with the network included; providers, precision, concurrency, routes and dates differ, retained and new timings can span sessions, and quick failures can reduce all-primary latency. The source published a separate cost ledger on 1 November 2026, stored as decision-reference-v2-cost-per-1000, which supersedes its earlier “cost not consolidated” note. Older 1,342-only scores are not converted into this bank. Not ranked here. Checked / reported: 2026-11-02 · Samples: 43 Snapshot: 2026-11-01 · Revision: f5fa9cfafccd97fcfbdb0336e14284faaa8d2d35 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
System 1 Decision Benchmark v2 · F02 · Forward conditionals	97.826%	Setup & source Forward conditionals: 46 questions. Nimble v4 Max; Hosted native; returned identities {"nimble-v4-max": 1933, "unreported in selected primary": 1}; collected 2026-10-16 to 2026-10-31 America/Toronto; one primary response per input, no scored retries. Unsuccessful responses counted as wrong: {"other provider failure": 1}. Development benchmark, not an independently sampled test: 1,934 questions in 41 categories, the frozen 1,342-question bank plus 192 logical and 400 practical additions. The bank was built with model-guided development, and Jev plus retained Microsoft and Drex feedback are development exposure. Related variants and a single recorded primary response per input limit generalization. The scoring key stays private, so public checks cover coverage, timing and aggregate consistency but cannot rescore answers. Strict accuracy counts a wrong usable answer and an unsuccessful response in the same fixed denominator. Timing pools primary client times with the network included; providers, precision, concurrency, routes and dates differ, retained and new timings can span sessions, and quick failures can reduce all-primary latency. The source published a separate cost ledger on 1 November 2026, stored as decision-reference-v2-cost-per-1000, which supersedes its earlier “cost not consolidated” note. Older 1,342-only scores are not converted into this bank. Not ranked here. Checked / reported: 2026-11-02 · Samples: 46 Snapshot: 2026-11-01 · Revision: f5fa9cfafccd97fcfbdb0336e14284faaa8d2d35 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
System 1 Decision Benchmark v2 · F03 · Necessary conditions	94.231%	Setup & source Necessary conditions: 52 questions. Nimble v4 Max; Hosted native; returned identities {"nimble-v4-max": 1933, "unreported in selected primary": 1}; collected 2026-10-16 to 2026-10-31 America/Toronto; one primary response per input, no scored retries. Unsuccessful responses counted as wrong: {"other provider failure": 1}. Development benchmark, not an independently sampled test: 1,934 questions in 41 categories, the frozen 1,342-question bank plus 192 logical and 400 practical additions. The bank was built with model-guided development, and Jev plus retained Microsoft and Drex feedback are development exposure. Related variants and a single recorded primary response per input limit generalization. The scoring key stays private, so public checks cover coverage, timing and aggregate consistency but cannot rescore answers. Strict accuracy counts a wrong usable answer and an unsuccessful response in the same fixed denominator. Timing pools primary client times with the network included; providers, precision, concurrency, routes and dates differ, retained and new timings can span sessions, and quick failures can reduce all-primary latency. The source published a separate cost ledger on 1 November 2026, stored as decision-reference-v2-cost-per-1000, which supersedes its earlier “cost not consolidated” note. Older 1,342-only scores are not converted into this bank. Not ranked here. Checked / reported: 2026-11-02 · Samples: 52 Snapshot: 2026-11-01 · Revision: f5fa9cfafccd97fcfbdb0336e14284faaa8d2d35 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
System 1 Decision Benchmark v2 · F04 · Boolean scope	92.593%	Setup & source Boolean scope: 54 questions. Nimble v4 Max; Hosted native; returned identities {"nimble-v4-max": 1933, "unreported in selected primary": 1}; collected 2026-10-16 to 2026-10-31 America/Toronto; one primary response per input, no scored retries. Unsuccessful responses counted as wrong: {"other provider failure": 1}. Development benchmark, not an independently sampled test: 1,934 questions in 41 categories, the frozen 1,342-question bank plus 192 logical and 400 practical additions. The bank was built with model-guided development, and Jev plus retained Microsoft and Drex feedback are development exposure. Related variants and a single recorded primary response per input limit generalization. The scoring key stays private, so public checks cover coverage, timing and aggregate consistency but cannot rescore answers. Strict accuracy counts a wrong usable answer and an unsuccessful response in the same fixed denominator. Timing pools primary client times with the network included; providers, precision, concurrency, routes and dates differ, retained and new timings can span sessions, and quick failures can reduce all-primary latency. The source published a separate cost ledger on 1 November 2026, stored as decision-reference-v2-cost-per-1000, which supersedes its earlier “cost not consolidated” note. Older 1,342-only scores are not converted into this bank. Not ranked here. Checked / reported: 2026-11-02 · Samples: 54 Snapshot: 2026-11-01 · Revision: f5fa9cfafccd97fcfbdb0336e14284faaa8d2d35 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
System 1 Decision Benchmark v2 · F05 · Quantifiers and sets	91.837%	Setup & source Quantifiers and sets: 49 questions. Nimble v4 Max; Hosted native; returned identities {"nimble-v4-max": 1933, "unreported in selected primary": 1}; collected 2026-10-16 to 2026-10-31 America/Toronto; one primary response per input, no scored retries. Unsuccessful responses counted as wrong: {"other provider failure": 1}. Development benchmark, not an independently sampled test: 1,934 questions in 41 categories, the frozen 1,342-question bank plus 192 logical and 400 practical additions. The bank was built with model-guided development, and Jev plus retained Microsoft and Drex feedback are development exposure. Related variants and a single recorded primary response per input limit generalization. The scoring key stays private, so public checks cover coverage, timing and aggregate consistency but cannot rescore answers. Strict accuracy counts a wrong usable answer and an unsuccessful response in the same fixed denominator. Timing pools primary client times with the network included; providers, precision, concurrency, routes and dates differ, retained and new timings can span sessions, and quick failures can reduce all-primary latency. The source published a separate cost ledger on 1 November 2026, stored as decision-reference-v2-cost-per-1000, which supersedes its earlier “cost not consolidated” note. Older 1,342-only scores are not converted into this bank. Not ranked here. Checked / reported: 2026-11-02 · Samples: 49 Snapshot: 2026-11-01 · Revision: f5fa9cfafccd97fcfbdb0336e14284faaa8d2d35 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
System 1 Decision Benchmark v2 · F06 · Comparisons	95.000%	Setup & source Comparisons: 40 questions. Nimble v4 Max; Hosted native; returned identities {"nimble-v4-max": 1933, "unreported in selected primary": 1}; collected 2026-10-16 to 2026-10-31 America/Toronto; one primary response per input, no scored retries. Unsuccessful responses counted as wrong: {"other provider failure": 1}. Development benchmark, not an independently sampled test: 1,934 questions in 41 categories, the frozen 1,342-question bank plus 192 logical and 400 practical additions. The bank was built with model-guided development, and Jev plus retained Microsoft and Drex feedback are development exposure. Related variants and a single recorded primary response per input limit generalization. The scoring key stays private, so public checks cover coverage, timing and aggregate consistency but cannot rescore answers. Strict accuracy counts a wrong usable answer and an unsuccessful response in the same fixed denominator. Timing pools primary client times with the network included; providers, precision, concurrency, routes and dates differ, retained and new timings can span sessions, and quick failures can reduce all-primary latency. The source published a separate cost ledger on 1 November 2026, stored as decision-reference-v2-cost-per-1000, which supersedes its earlier “cost not consolidated” note. Older 1,342-only scores are not converted into this bank. Not ranked here. Checked / reported: 2026-11-02 · Samples: 40 Snapshot: 2026-11-01 · Revision: f5fa9cfafccd97fcfbdb0336e14284faaa8d2d35 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
System 1 Decision Benchmark v2 · F07 · Scheduling	67.105%	Setup & source Scheduling: 76 questions. Nimble v4 Max; Hosted native; returned identities {"nimble-v4-max": 1933, "unreported in selected primary": 1}; collected 2026-10-16 to 2026-10-31 America/Toronto; one primary response per input, no scored retries. Unsuccessful responses counted as wrong: {"other provider failure": 1}. Development benchmark, not an independently sampled test: 1,934 questions in 41 categories, the frozen 1,342-question bank plus 192 logical and 400 practical additions. The bank was built with model-guided development, and Jev plus retained Microsoft and Drex feedback are development exposure. Related variants and a single recorded primary response per input limit generalization. The scoring key stays private, so public checks cover coverage, timing and aggregate consistency but cannot rescore answers. Strict accuracy counts a wrong usable answer and an unsuccessful response in the same fixed denominator. Timing pools primary client times with the network included; providers, precision, concurrency, routes and dates differ, retained and new timings can span sessions, and quick failures can reduce all-primary latency. The source published a separate cost ledger on 1 November 2026, stored as decision-reference-v2-cost-per-1000, which supersedes its earlier “cost not consolidated” note. Older 1,342-only scores are not converted into this bank. Not ranked here. Checked / reported: 2026-11-02 · Samples: 76 Snapshot: 2026-11-01 · Revision: f5fa9cfafccd97fcfbdb0336e14284faaa8d2d35 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
System 1 Decision Benchmark v2 · F08 · One-to-one assignments	89.552%	Setup & source One-to-one assignments: 67 questions. Nimble v4 Max; Hosted native; returned identities {"nimble-v4-max": 1933, "unreported in selected primary": 1}; collected 2026-10-16 to 2026-10-31 America/Toronto; one primary response per input, no scored retries. Unsuccessful responses counted as wrong: {"other provider failure": 1}. Development benchmark, not an independently sampled test: 1,934 questions in 41 categories, the frozen 1,342-question bank plus 192 logical and 400 practical additions. The bank was built with model-guided development, and Jev plus retained Microsoft and Drex feedback are development exposure. Related variants and a single recorded primary response per input limit generalization. The scoring key stays private, so public checks cover coverage, timing and aggregate consistency but cannot rescore answers. Strict accuracy counts a wrong usable answer and an unsuccessful response in the same fixed denominator. Timing pools primary client times with the network included; providers, precision, concurrency, routes and dates differ, retained and new timings can span sessions, and quick failures can reduce all-primary latency. The source published a separate cost ledger on 1 November 2026, stored as decision-reference-v2-cost-per-1000, which supersedes its earlier “cost not consolidated” note. Older 1,342-only scores are not converted into this bank. Not ranked here. Checked / reported: 2026-11-02 · Samples: 67 Snapshot: 2026-11-01 · Revision: f5fa9cfafccd97fcfbdb0336e14284faaa8d2d35 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
System 1 Decision Benchmark v2 · F09 · Exact truth counting	54.930%	Setup & source Exact truth counting: 71 questions. Nimble v4 Max; Hosted native; returned identities {"nimble-v4-max": 1933, "unreported in selected primary": 1}; collected 2026-10-16 to 2026-10-31 America/Toronto; one primary response per input, no scored retries. Unsuccessful responses counted as wrong: {"other provider failure": 1}. Development benchmark, not an independently sampled test: 1,934 questions in 41 categories, the frozen 1,342-question bank plus 192 logical and 400 practical additions. The bank was built with model-guided development, and Jev plus retained Microsoft and Drex feedback are development exposure. Related variants and a single recorded primary response per input limit generalization. The scoring key stays private, so public checks cover coverage, timing and aggregate consistency but cannot rescore answers. Strict accuracy counts a wrong usable answer and an unsuccessful response in the same fixed denominator. Timing pools primary client times with the network included; providers, precision, concurrency, routes and dates differ, retained and new timings can span sessions, and quick failures can reduce all-primary latency. The source published a separate cost ledger on 1 November 2026, stored as decision-reference-v2-cost-per-1000, which supersedes its earlier “cost not consolidated” note. Older 1,342-only scores are not converted into this bank. Not ranked here. Checked / reported: 2026-11-02 · Samples: 71 Snapshot: 2026-11-01 · Revision: f5fa9cfafccd97fcfbdb0336e14284faaa8d2d35 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
System 1 Decision Benchmark v2 · F10 · Truth-teller consistency	71.622%	Setup & source Truth-teller consistency: 74 questions. Nimble v4 Max; Hosted native; returned identities {"nimble-v4-max": 1933, "unreported in selected primary": 1}; collected 2026-10-16 to 2026-10-31 America/Toronto; one primary response per input, no scored retries. Unsuccessful responses counted as wrong: {"other provider failure": 1}. Development benchmark, not an independently sampled test: 1,934 questions in 41 categories, the frozen 1,342-question bank plus 192 logical and 400 practical additions. The bank was built with model-guided development, and Jev plus retained Microsoft and Drex feedback are development exposure. Related variants and a single recorded primary response per input limit generalization. The scoring key stays private, so public checks cover coverage, timing and aggregate consistency but cannot rescore answers. Strict accuracy counts a wrong usable answer and an unsuccessful response in the same fixed denominator. Timing pools primary client times with the network included; providers, precision, concurrency, routes and dates differ, retained and new timings can span sessions, and quick failures can reduce all-primary latency. The source published a separate cost ledger on 1 November 2026, stored as decision-reference-v2-cost-per-1000, which supersedes its earlier “cost not consolidated” note. Older 1,342-only scores are not converted into this bank. Not ranked here. Checked / reported: 2026-11-02 · Samples: 74 Snapshot: 2026-11-01 · Revision: f5fa9cfafccd97fcfbdb0336e14284faaa8d2d35 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
System 1 Decision Benchmark v2 · F11 · Sufficiency and inconsistency	78.049%	Setup & source Sufficiency and inconsistency: 41 questions. Nimble v4 Max; Hosted native; returned identities {"nimble-v4-max": 1933, "unreported in selected primary": 1}; collected 2026-10-16 to 2026-10-31 America/Toronto; one primary response per input, no scored retries. Unsuccessful responses counted as wrong: {"other provider failure": 1}. Development benchmark, not an independently sampled test: 1,934 questions in 41 categories, the frozen 1,342-question bank plus 192 logical and 400 practical additions. The bank was built with model-guided development, and Jev plus retained Microsoft and Drex feedback are development exposure. Related variants and a single recorded primary response per input limit generalization. The scoring key stays private, so public checks cover coverage, timing and aggregate consistency but cannot rescore answers. Strict accuracy counts a wrong usable answer and an unsuccessful response in the same fixed denominator. Timing pools primary client times with the network included; providers, precision, concurrency, routes and dates differ, retained and new timings can span sessions, and quick failures can reduce all-primary latency. The source published a separate cost ledger on 1 November 2026, stored as decision-reference-v2-cost-per-1000, which supersedes its earlier “cost not consolidated” note. Older 1,342-only scores are not converted into this bank. Not ranked here. Checked / reported: 2026-11-02 · Samples: 41 Snapshot: 2026-11-01 · Revision: f5fa9cfafccd97fcfbdb0336e14284faaa8d2d35 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
System 1 Decision Benchmark v2 · F12 · Object swaps	67.308%	Setup & source Object swaps: 52 questions. Nimble v4 Max; Hosted native; returned identities {"nimble-v4-max": 1933, "unreported in selected primary": 1}; collected 2026-10-16 to 2026-10-31 America/Toronto; one primary response per input, no scored retries. Unsuccessful responses counted as wrong: {"other provider failure": 1}. Development benchmark, not an independently sampled test: 1,934 questions in 41 categories, the frozen 1,342-question bank plus 192 logical and 400 practical additions. The bank was built with model-guided development, and Jev plus retained Microsoft and Drex feedback are development exposure. Related variants and a single recorded primary response per input limit generalization. The scoring key stays private, so public checks cover coverage, timing and aggregate consistency but cannot rescore answers. Strict accuracy counts a wrong usable answer and an unsuccessful response in the same fixed denominator. Timing pools primary client times with the network included; providers, precision, concurrency, routes and dates differ, retained and new timings can span sessions, and quick failures can reduce all-primary latency. The source published a separate cost ledger on 1 November 2026, stored as decision-reference-v2-cost-per-1000, which supersedes its earlier “cost not consolidated” note. Older 1,342-only scores are not converted into this bank. Not ranked here. Checked / reported: 2026-11-02 · Samples: 52 Snapshot: 2026-11-01 · Revision: f5fa9cfafccd97fcfbdb0336e14284faaa8d2d35 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
System 1 Decision Benchmark v2 · F13 · Sequential procedures	39.583%	Setup & source Sequential procedures: 48 questions. Nimble v4 Max; Hosted native; returned identities {"nimble-v4-max": 1933, "unreported in selected primary": 1}; collected 2026-10-16 to 2026-10-31 America/Toronto; one primary response per input, no scored retries. Unsuccessful responses counted as wrong: {"other provider failure": 1}. Development benchmark, not an independently sampled test: 1,934 questions in 41 categories, the frozen 1,342-question bank plus 192 logical and 400 practical additions. The bank was built with model-guided development, and Jev plus retained Microsoft and Drex feedback are development exposure. Related variants and a single recorded primary response per input limit generalization. The scoring key stays private, so public checks cover coverage, timing and aggregate consistency but cannot rescore answers. Strict accuracy counts a wrong usable answer and an unsuccessful response in the same fixed denominator. Timing pools primary client times with the network included; providers, precision, concurrency, routes and dates differ, retained and new timings can span sessions, and quick failures can reduce all-primary latency. The source published a separate cost ledger on 1 November 2026, stored as decision-reference-v2-cost-per-1000, which supersedes its earlier “cost not consolidated” note. Older 1,342-only scores are not converted into this bank. Not ranked here. Checked / reported: 2026-11-02 · Samples: 48 Snapshot: 2026-11-01 · Revision: f5fa9cfafccd97fcfbdb0336e14284faaa8d2d35 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
System 1 Decision Benchmark v2 · F14 · Temporal intervals	97.619%	Setup & source Temporal intervals: 42 questions. Nimble v4 Max; Hosted native; returned identities {"nimble-v4-max": 1933, "unreported in selected primary": 1}; collected 2026-10-16 to 2026-10-31 America/Toronto; one primary response per input, no scored retries. Unsuccessful responses counted as wrong: {"other provider failure": 1}. Development benchmark, not an independently sampled test: 1,934 questions in 41 categories, the frozen 1,342-question bank plus 192 logical and 400 practical additions. The bank was built with model-guided development, and Jev plus retained Microsoft and Drex feedback are development exposure. Related variants and a single recorded primary response per input limit generalization. The scoring key stays private, so public checks cover coverage, timing and aggregate consistency but cannot rescore answers. Strict accuracy counts a wrong usable answer and an unsuccessful response in the same fixed denominator. Timing pools primary client times with the network included; providers, precision, concurrency, routes and dates differ, retained and new timings can span sessions, and quick failures can reduce all-primary latency. The source published a separate cost ledger on 1 November 2026, stored as decision-reference-v2-cost-per-1000, which supersedes its earlier “cost not consolidated” note. Older 1,342-only scores are not converted into this bank. Not ranked here. Checked / reported: 2026-11-02 · Samples: 42 Snapshot: 2026-11-01 · Revision: f5fa9cfafccd97fcfbdb0336e14284faaa8d2d35 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
System 1 Decision Benchmark v2 · F15 · Spatial tracking	73.469%	Setup & source Spatial tracking: 49 questions. Nimble v4 Max; Hosted native; returned identities {"nimble-v4-max": 1933, "unreported in selected primary": 1}; collected 2026-10-16 to 2026-10-31 America/Toronto; one primary response per input, no scored retries. Unsuccessful responses counted as wrong: {"other provider failure": 1}. Development benchmark, not an independently sampled test: 1,934 questions in 41 categories, the frozen 1,342-question bank plus 192 logical and 400 practical additions. The bank was built with model-guided development, and Jev plus retained Microsoft and Drex feedback are development exposure. Related variants and a single recorded primary response per input limit generalization. The scoring key stays private, so public checks cover coverage, timing and aggregate consistency but cannot rescore answers. Strict accuracy counts a wrong usable answer and an unsuccessful response in the same fixed denominator. Timing pools primary client times with the network included; providers, precision, concurrency, routes and dates differ, retained and new timings can span sessions, and quick failures can reduce all-primary latency. The source published a separate cost ledger on 1 November 2026, stored as decision-reference-v2-cost-per-1000, which supersedes its earlier “cost not consolidated” note. Older 1,342-only scores are not converted into this bank. Not ranked here. Checked / reported: 2026-11-02 · Samples: 49 Snapshot: 2026-11-01 · Revision: f5fa9cfafccd97fcfbdb0336e14284faaa8d2d35 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
System 1 Decision Benchmark v2 · F16 · Reference binding	76.000%	Setup & source Reference binding: 50 questions. Nimble v4 Max; Hosted native; returned identities {"nimble-v4-max": 1933, "unreported in selected primary": 1}; collected 2026-10-16 to 2026-10-31 America/Toronto; one primary response per input, no scored retries. Unsuccessful responses counted as wrong: {"other provider failure": 1}. Development benchmark, not an independently sampled test: 1,934 questions in 41 categories, the frozen 1,342-question bank plus 192 logical and 400 practical additions. The bank was built with model-guided development, and Jev plus retained Microsoft and Drex feedback are development exposure. Related variants and a single recorded primary response per input limit generalization. The scoring key stays private, so public checks cover coverage, timing and aggregate consistency but cannot rescore answers. Strict accuracy counts a wrong usable answer and an unsuccessful response in the same fixed denominator. Timing pools primary client times with the network included; providers, precision, concurrency, routes and dates differ, retained and new timings can span sessions, and quick failures can reduce all-primary latency. The source published a separate cost ledger on 1 November 2026, stored as decision-reference-v2-cost-per-1000, which supersedes its earlier “cost not consolidated” note. Older 1,342-only scores are not converted into this bank. Not ranked here. Checked / reported: 2026-11-02 · Samples: 50 Snapshot: 2026-11-01 · Revision: f5fa9cfafccd97fcfbdb0336e14284faaa8d2d35 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
System 1 Decision Benchmark v2 · F17 · Beliefs and perspectives	92.000%	Setup & source Beliefs and perspectives: 50 questions. Nimble v4 Max; Hosted native; returned identities {"nimble-v4-max": 1933, "unreported in selected primary": 1}; collected 2026-10-16 to 2026-10-31 America/Toronto; one primary response per input, no scored retries. Unsuccessful responses counted as wrong: {"other provider failure": 1}. Development benchmark, not an independently sampled test: 1,934 questions in 41 categories, the frozen 1,342-question bank plus 192 logical and 400 practical additions. The bank was built with model-guided development, and Jev plus retained Microsoft and Drex feedback are development exposure. Related variants and a single recorded primary response per input limit generalization. The scoring key stays private, so public checks cover coverage, timing and aggregate consistency but cannot rescore answers. Strict accuracy counts a wrong usable answer and an unsuccessful response in the same fixed denominator. Timing pools primary client times with the network included; providers, precision, concurrency, routes and dates differ, retained and new timings can span sessions, and quick failures can reduce all-primary latency. The source published a separate cost ledger on 1 November 2026, stored as decision-reference-v2-cost-per-1000, which supersedes its earlier “cost not consolidated” note. Older 1,342-only scores are not converted into this bank. Not ranked here. Checked / reported: 2026-11-02 · Samples: 50 Snapshot: 2026-11-01 · Revision: f5fa9cfafccd97fcfbdb0336e14284faaa8d2d35 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
System 1 Decision Benchmark v2 · F18 · Causal intervention	90.000%	Setup & source Causal intervention: 50 questions. Nimble v4 Max; Hosted native; returned identities {"nimble-v4-max": 1933, "unreported in selected primary": 1}; collected 2026-10-16 to 2026-10-31 America/Toronto; one primary response per input, no scored retries. Unsuccessful responses counted as wrong: {"other provider failure": 1}. Development benchmark, not an independently sampled test: 1,934 questions in 41 categories, the frozen 1,342-question bank plus 192 logical and 400 practical additions. The bank was built with model-guided development, and Jev plus retained Microsoft and Drex feedback are development exposure. Related variants and a single recorded primary response per input limit generalization. The scoring key stays private, so public checks cover coverage, timing and aggregate consistency but cannot rescore answers. Strict accuracy counts a wrong usable answer and an unsuccessful response in the same fixed denominator. Timing pools primary client times with the network included; providers, precision, concurrency, routes and dates differ, retained and new timings can span sessions, and quick failures can reduce all-primary latency. The source published a separate cost ledger on 1 November 2026, stored as decision-reference-v2-cost-per-1000, which supersedes its earlier “cost not consolidated” note. Older 1,342-only scores are not converted into this bank. Not ranked here. Checked / reported: 2026-11-02 · Samples: 50 Snapshot: 2026-11-01 · Revision: f5fa9cfafccd97fcfbdb0336e14284faaa8d2d35 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
System 1 Decision Benchmark v2 · F19 · Exceptions and priorities	98.276%	Setup & source Exceptions and priorities: 58 questions. Nimble v4 Max; Hosted native; returned identities {"nimble-v4-max": 1933, "unreported in selected primary": 1}; collected 2026-10-16 to 2026-10-31 America/Toronto; one primary response per input, no scored retries. Unsuccessful responses counted as wrong: {"other provider failure": 1}. Development benchmark, not an independently sampled test: 1,934 questions in 41 categories, the frozen 1,342-question bank plus 192 logical and 400 practical additions. The bank was built with model-guided development, and Jev plus retained Microsoft and Drex feedback are development exposure. Related variants and a single recorded primary response per input limit generalization. The scoring key stays private, so public checks cover coverage, timing and aggregate consistency but cannot rescore answers. Strict accuracy counts a wrong usable answer and an unsuccessful response in the same fixed denominator. Timing pools primary client times with the network included; providers, precision, concurrency, routes and dates differ, retained and new timings can span sessions, and quick failures can reduce all-primary latency. The source published a separate cost ledger on 1 November 2026, stored as decision-reference-v2-cost-per-1000, which supersedes its earlier “cost not consolidated” note. Older 1,342-only scores are not converted into this bank. Not ranked here. Checked / reported: 2026-11-02 · Samples: 58 Snapshot: 2026-11-01 · Revision: f5fa9cfafccd97fcfbdb0336e14284faaa8d2d35 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
System 1 Decision Benchmark v2 · F20 · Planning	93.023%	Setup & source Planning: 43 questions. Nimble v4 Max; Hosted native; returned identities {"nimble-v4-max": 1933, "unreported in selected primary": 1}; collected 2026-10-16 to 2026-10-31 America/Toronto; one primary response per input, no scored retries. Unsuccessful responses counted as wrong: {"other provider failure": 1}. Development benchmark, not an independently sampled test: 1,934 questions in 41 categories, the frozen 1,342-question bank plus 192 logical and 400 practical additions. The bank was built with model-guided development, and Jev plus retained Microsoft and Drex feedback are development exposure. Related variants and a single recorded primary response per input limit generalization. The scoring key stays private, so public checks cover coverage, timing and aggregate consistency but cannot rescore answers. Strict accuracy counts a wrong usable answer and an unsuccessful response in the same fixed denominator. Timing pools primary client times with the network included; providers, precision, concurrency, routes and dates differ, retained and new timings can span sessions, and quick failures can reduce all-primary latency. The source published a separate cost ledger on 1 November 2026, stored as decision-reference-v2-cost-per-1000, which supersedes its earlier “cost not consolidated” note. Older 1,342-only scores are not converted into this bank. Not ranked here. Checked / reported: 2026-11-02 · Samples: 43 Snapshot: 2026-11-01 · Revision: f5fa9cfafccd97fcfbdb0336e14284faaa8d2d35 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
System 1 Decision Benchmark v2 · F21 · Directed reachability	93.023%	Setup & source Directed reachability: 43 questions. Nimble v4 Max; Hosted native; returned identities {"nimble-v4-max": 1933, "unreported in selected primary": 1}; collected 2026-10-16 to 2026-10-31 America/Toronto; one primary response per input, no scored retries. Unsuccessful responses counted as wrong: {"other provider failure": 1}. Development benchmark, not an independently sampled test: 1,934 questions in 41 categories, the frozen 1,342-question bank plus 192 logical and 400 practical additions. The bank was built with model-guided development, and Jev plus retained Microsoft and Drex feedback are development exposure. Related variants and a single recorded primary response per input limit generalization. The scoring key stays private, so public checks cover coverage, timing and aggregate consistency but cannot rescore answers. Strict accuracy counts a wrong usable answer and an unsuccessful response in the same fixed denominator. Timing pools primary client times with the network included; providers, precision, concurrency, routes and dates differ, retained and new timings can span sessions, and quick failures can reduce all-primary latency. The source published a separate cost ledger on 1 November 2026, stored as decision-reference-v2-cost-per-1000, which supersedes its earlier “cost not consolidated” note. Older 1,342-only scores are not converted into this bank. Not ranked here. Checked / reported: 2026-11-02 · Samples: 43 Snapshot: 2026-11-01 · Revision: f5fa9cfafccd97fcfbdb0336e14284faaa8d2d35 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
System 1 Decision Benchmark v2 · F22 · Preference ordering	96.078%	Setup & source Preference ordering: 51 questions. Nimble v4 Max; Hosted native; returned identities {"nimble-v4-max": 1933, "unreported in selected primary": 1}; collected 2026-10-16 to 2026-10-31 America/Toronto; one primary response per input, no scored retries. Unsuccessful responses counted as wrong: {"other provider failure": 1}. Development benchmark, not an independently sampled test: 1,934 questions in 41 categories, the frozen 1,342-question bank plus 192 logical and 400 practical additions. The bank was built with model-guided development, and Jev plus retained Microsoft and Drex feedback are development exposure. Related variants and a single recorded primary response per input limit generalization. The scoring key stays private, so public checks cover coverage, timing and aggregate consistency but cannot rescore answers. Strict accuracy counts a wrong usable answer and an unsuccessful response in the same fixed denominator. Timing pools primary client times with the network included; providers, precision, concurrency, routes and dates differ, retained and new timings can span sessions, and quick failures can reduce all-primary latency. The source published a separate cost ledger on 1 November 2026, stored as decision-reference-v2-cost-per-1000, which supersedes its earlier “cost not consolidated” note. Older 1,342-only scores are not converted into this bank. Not ranked here. Checked / reported: 2026-11-02 · Samples: 51 Snapshot: 2026-11-01 · Revision: f5fa9cfafccd97fcfbdb0336e14284faaa8d2d35 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
System 1 Decision Benchmark v2 · F23 · Rule counterexamples	100.000%	Setup & source Rule counterexamples: 52 questions. Nimble v4 Max; Hosted native; returned identities {"nimble-v4-max": 1933, "unreported in selected primary": 1}; collected 2026-10-16 to 2026-10-31 America/Toronto; one primary response per input, no scored retries. Unsuccessful responses counted as wrong: {"other provider failure": 1}. Development benchmark, not an independently sampled test: 1,934 questions in 41 categories, the frozen 1,342-question bank plus 192 logical and 400 practical additions. The bank was built with model-guided development, and Jev plus retained Microsoft and Drex feedback are development exposure. Related variants and a single recorded primary response per input limit generalization. The scoring key stays private, so public checks cover coverage, timing and aggregate consistency but cannot rescore answers. Strict accuracy counts a wrong usable answer and an unsuccessful response in the same fixed denominator. Timing pools primary client times with the network included; providers, precision, concurrency, routes and dates differ, retained and new timings can span sessions, and quick failures can reduce all-primary latency. The source published a separate cost ledger on 1 November 2026, stored as decision-reference-v2-cost-per-1000, which supersedes its earlier “cost not consolidated” note. Older 1,342-only scores are not converted into this bank. Not ranked here. Checked / reported: 2026-11-02 · Samples: 52 Snapshot: 2026-11-01 · Revision: f5fa9cfafccd97fcfbdb0336e14284faaa8d2d35 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
System 1 Decision Benchmark v2 · F24 · Authority and stale records	97.674%	Setup & source Authority and stale records: 43 questions. Nimble v4 Max; Hosted native; returned identities {"nimble-v4-max": 1933, "unreported in selected primary": 1}; collected 2026-10-16 to 2026-10-31 America/Toronto; one primary response per input, no scored retries. Unsuccessful responses counted as wrong: {"other provider failure": 1}. Development benchmark, not an independently sampled test: 1,934 questions in 41 categories, the frozen 1,342-question bank plus 192 logical and 400 practical additions. The bank was built with model-guided development, and Jev plus retained Microsoft and Drex feedback are development exposure. Related variants and a single recorded primary response per input limit generalization. The scoring key stays private, so public checks cover coverage, timing and aggregate consistency but cannot rescore answers. Strict accuracy counts a wrong usable answer and an unsuccessful response in the same fixed denominator. Timing pools primary client times with the network included; providers, precision, concurrency, routes and dates differ, retained and new timings can span sessions, and quick failures can reduce all-primary latency. The source published a separate cost ledger on 1 November 2026, stored as decision-reference-v2-cost-per-1000, which supersedes its earlier “cost not consolidated” note. Older 1,342-only scores are not converted into this bank. Not ranked here. Checked / reported: 2026-11-02 · Samples: 43 Snapshot: 2026-11-01 · Revision: f5fa9cfafccd97fcfbdb0336e14284faaa8d2d35 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
System 1 Decision Benchmark v2 · F25 · Fault diagnosis	94.000%	Setup & source Fault diagnosis: 50 questions. Nimble v4 Max; Hosted native; returned identities {"nimble-v4-max": 1933, "unreported in selected primary": 1}; collected 2026-10-16 to 2026-10-31 America/Toronto; one primary response per input, no scored retries. Unsuccessful responses counted as wrong: {"other provider failure": 1}. Development benchmark, not an independently sampled test: 1,934 questions in 41 categories, the frozen 1,342-question bank plus 192 logical and 400 practical additions. The bank was built with model-guided development, and Jev plus retained Microsoft and Drex feedback are development exposure. Related variants and a single recorded primary response per input limit generalization. The scoring key stays private, so public checks cover coverage, timing and aggregate consistency but cannot rescore answers. Strict accuracy counts a wrong usable answer and an unsuccessful response in the same fixed denominator. Timing pools primary client times with the network included; providers, precision, concurrency, routes and dates differ, retained and new timings can span sessions, and quick failures can reduce all-primary latency. The source published a separate cost ledger on 1 November 2026, stored as decision-reference-v2-cost-per-1000, which supersedes its earlier “cost not consolidated” note. Older 1,342-only scores are not converted into this bank. Not ranked here. Checked / reported: 2026-11-02 · Samples: 50 Snapshot: 2026-11-01 · Revision: f5fa9cfafccd97fcfbdb0336e14284faaa8d2d35 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
System 1 Decision Benchmark v2 · F26 · Probabilistic risk	62.500%	Setup & source Probabilistic risk: 40 questions. Nimble v4 Max; Hosted native; returned identities {"nimble-v4-max": 1933, "unreported in selected primary": 1}; collected 2026-10-16 to 2026-10-31 America/Toronto; one primary response per input, no scored retries. Unsuccessful responses counted as wrong: {"other provider failure": 1}. Development benchmark, not an independently sampled test: 1,934 questions in 41 categories, the frozen 1,342-question bank plus 192 logical and 400 practical additions. The bank was built with model-guided development, and Jev plus retained Microsoft and Drex feedback are development exposure. Related variants and a single recorded primary response per input limit generalization. The scoring key stays private, so public checks cover coverage, timing and aggregate consistency but cannot rescore answers. Strict accuracy counts a wrong usable answer and an unsuccessful response in the same fixed denominator. Timing pools primary client times with the network included; providers, precision, concurrency, routes and dates differ, retained and new timings can span sessions, and quick failures can reduce all-primary latency. The source published a separate cost ledger on 1 November 2026, stored as decision-reference-v2-cost-per-1000, which supersedes its earlier “cost not consolidated” note. Older 1,342-only scores are not converted into this bank. Not ranked here. Checked / reported: 2026-11-02 · Samples: 40 Snapshot: 2026-11-01 · Revision: f5fa9cfafccd97fcfbdb0336e14284faaa8d2d35 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
System 1 Decision Benchmark v2 · F27 · Value of information	62.500%	Setup & source Value of information: 40 questions. Nimble v4 Max; Hosted native; returned identities {"nimble-v4-max": 1933, "unreported in selected primary": 1}; collected 2026-10-16 to 2026-10-31 America/Toronto; one primary response per input, no scored retries. Unsuccessful responses counted as wrong: {"other provider failure": 1}. Development benchmark, not an independently sampled test: 1,934 questions in 41 categories, the frozen 1,342-question bank plus 192 logical and 400 practical additions. The bank was built with model-guided development, and Jev plus retained Microsoft and Drex feedback are development exposure. Related variants and a single recorded primary response per input limit generalization. The scoring key stays private, so public checks cover coverage, timing and aggregate consistency but cannot rescore answers. Strict accuracy counts a wrong usable answer and an unsuccessful response in the same fixed denominator. Timing pools primary client times with the network included; providers, precision, concurrency, routes and dates differ, retained and new timings can span sessions, and quick failures can reduce all-primary latency. The source published a separate cost ledger on 1 November 2026, stored as decision-reference-v2-cost-per-1000, which supersedes its earlier “cost not consolidated” note. Older 1,342-only scores are not converted into this bank. Not ranked here. Checked / reported: 2026-11-02 · Samples: 40 Snapshot: 2026-11-01 · Revision: f5fa9cfafccd97fcfbdb0336e14284faaa8d2d35 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
System 1 Decision Benchmark v2 · F28 · Robust decisions without a prior	47.500%	Setup & source Robust decisions without a prior: 40 questions. Nimble v4 Max; Hosted native; returned identities {"nimble-v4-max": 1933, "unreported in selected primary": 1}; collected 2026-10-16 to 2026-10-31 America/Toronto; one primary response per input, no scored retries. Unsuccessful responses counted as wrong: {"other provider failure": 1}. Development benchmark, not an independently sampled test: 1,934 questions in 41 categories, the frozen 1,342-question bank plus 192 logical and 400 practical additions. The bank was built with model-guided development, and Jev plus retained Microsoft and Drex feedback are development exposure. Related variants and a single recorded primary response per input limit generalization. The scoring key stays private, so public checks cover coverage, timing and aggregate consistency but cannot rescore answers. Strict accuracy counts a wrong usable answer and an unsuccessful response in the same fixed denominator. Timing pools primary client times with the network included; providers, precision, concurrency, routes and dates differ, retained and new timings can span sessions, and quick failures can reduce all-primary latency. The source published a separate cost ledger on 1 November 2026, stored as decision-reference-v2-cost-per-1000, which supersedes its earlier “cost not consolidated” note. Older 1,342-only scores are not converted into this bank. Not ranked here. Checked / reported: 2026-11-02 · Samples: 40 Snapshot: 2026-11-01 · Revision: f5fa9cfafccd97fcfbdb0336e14284faaa8d2d35 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
System 1 Decision Benchmark v2 · F29 · Strategic decisions	70.000%	Setup & source Strategic decisions: 40 questions. Nimble v4 Max; Hosted native; returned identities {"nimble-v4-max": 1933, "unreported in selected primary": 1}; collected 2026-10-16 to 2026-10-31 America/Toronto; one primary response per input, no scored retries. Unsuccessful responses counted as wrong: {"other provider failure": 1}. Development benchmark, not an independently sampled test: 1,934 questions in 41 categories, the frozen 1,342-question bank plus 192 logical and 400 practical additions. The bank was built with model-guided development, and Jev plus retained Microsoft and Drex feedback are development exposure. Related variants and a single recorded primary response per input limit generalization. The scoring key stays private, so public checks cover coverage, timing and aggregate consistency but cannot rescore answers. Strict accuracy counts a wrong usable answer and an unsuccessful response in the same fixed denominator. Timing pools primary client times with the network included; providers, precision, concurrency, routes and dates differ, retained and new timings can span sessions, and quick failures can reduce all-primary latency. The source published a separate cost ledger on 1 November 2026, stored as decision-reference-v2-cost-per-1000, which supersedes its earlier “cost not consolidated” note. Older 1,342-only scores are not converted into this bank. Not ranked here. Checked / reported: 2026-11-02 · Samples: 40 Snapshot: 2026-11-01 · Revision: f5fa9cfafccd97fcfbdb0336e14284faaa8d2d35 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
System 1 Decision Benchmark v2 · F30 · Concurrent-history consistency	57.500%	Setup & source Concurrent-history consistency: 40 questions. Nimble v4 Max; Hosted native; returned identities {"nimble-v4-max": 1933, "unreported in selected primary": 1}; collected 2026-10-16 to 2026-10-31 America/Toronto; one primary response per input, no scored retries. Unsuccessful responses counted as wrong: {"other provider failure": 1}. Development benchmark, not an independently sampled test: 1,934 questions in 41 categories, the frozen 1,342-question bank plus 192 logical and 400 practical additions. The bank was built with model-guided development, and Jev plus retained Microsoft and Drex feedback are development exposure. Related variants and a single recorded primary response per input limit generalization. The scoring key stays private, so public checks cover coverage, timing and aggregate consistency but cannot rescore answers. Strict accuracy counts a wrong usable answer and an unsuccessful response in the same fixed denominator. Timing pools primary client times with the network included; providers, precision, concurrency, routes and dates differ, retained and new timings can span sessions, and quick failures can reduce all-primary latency. The source published a separate cost ledger on 1 November 2026, stored as decision-reference-v2-cost-per-1000, which supersedes its earlier “cost not consolidated” note. Older 1,342-only scores are not converted into this bank. Not ranked here. Checked / reported: 2026-11-02 · Samples: 40 Snapshot: 2026-11-01 · Revision: f5fa9cfafccd97fcfbdb0336e14284faaa8d2d35 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
System 1 Decision Benchmark v2 · F31 · Information-flow security	75.000%	Setup & source Information-flow security: 40 questions. Nimble v4 Max; Hosted native; returned identities {"nimble-v4-max": 1933, "unreported in selected primary": 1}; collected 2026-10-16 to 2026-10-31 America/Toronto; one primary response per input, no scored retries. Unsuccessful responses counted as wrong: {"other provider failure": 1}. Development benchmark, not an independently sampled test: 1,934 questions in 41 categories, the frozen 1,342-question bank plus 192 logical and 400 practical additions. The bank was built with model-guided development, and Jev plus retained Microsoft and Drex feedback are development exposure. Related variants and a single recorded primary response per input limit generalization. The scoring key stays private, so public checks cover coverage, timing and aggregate consistency but cannot rescore answers. Strict accuracy counts a wrong usable answer and an unsuccessful response in the same fixed denominator. Timing pools primary client times with the network included; providers, precision, concurrency, routes and dates differ, retained and new timings can span sessions, and quick failures can reduce all-primary latency. The source published a separate cost ledger on 1 November 2026, stored as decision-reference-v2-cost-per-1000, which supersedes its earlier “cost not consolidated” note. Older 1,342-only scores are not converted into this bank. Not ranked here. Checked / reported: 2026-11-02 · Samples: 40 Snapshot: 2026-11-01 · Revision: f5fa9cfafccd97fcfbdb0336e14284faaa8d2d35 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
System 1 Decision Benchmark v2 · P01 · Understanding requests and routing	87.500%	Setup & source Understanding requests and routing: 40 questions. Nimble v4 Max; Hosted native; returned identities {"nimble-v4-max": 1933, "unreported in selected primary": 1}; collected 2026-10-16 to 2026-10-31 America/Toronto; one primary response per input, no scored retries. Unsuccessful responses counted as wrong: {"other provider failure": 1}. Development benchmark, not an independently sampled test: 1,934 questions in 41 categories, the frozen 1,342-question bank plus 192 logical and 400 practical additions. The bank was built with model-guided development, and Jev plus retained Microsoft and Drex feedback are development exposure. Related variants and a single recorded primary response per input limit generalization. The scoring key stays private, so public checks cover coverage, timing and aggregate consistency but cannot rescore answers. Strict accuracy counts a wrong usable answer and an unsuccessful response in the same fixed denominator. Timing pools primary client times with the network included; providers, precision, concurrency, routes and dates differ, retained and new timings can span sessions, and quick failures can reduce all-primary latency. The source published a separate cost ledger on 1 November 2026, stored as decision-reference-v2-cost-per-1000, which supersedes its earlier “cost not consolidated” note. Older 1,342-only scores are not converted into this bank. Not ranked here. Checked / reported: 2026-11-02 · Samples: 40 Snapshot: 2026-11-01 · Revision: f5fa9cfafccd97fcfbdb0336e14284faaa8d2d35 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
System 1 Decision Benchmark v2 · P02 · Recognizing multiple requested actions	97.500%	Setup & source Recognizing multiple requested actions: 40 questions. Nimble v4 Max; Hosted native; returned identities {"nimble-v4-max": 1933, "unreported in selected primary": 1}; collected 2026-10-16 to 2026-10-31 America/Toronto; one primary response per input, no scored retries. Unsuccessful responses counted as wrong: {"other provider failure": 1}. Development benchmark, not an independently sampled test: 1,934 questions in 41 categories, the frozen 1,342-question bank plus 192 logical and 400 practical additions. The bank was built with model-guided development, and Jev plus retained Microsoft and Drex feedback are development exposure. Related variants and a single recorded primary response per input limit generalization. The scoring key stays private, so public checks cover coverage, timing and aggregate consistency but cannot rescore answers. Strict accuracy counts a wrong usable answer and an unsuccessful response in the same fixed denominator. Timing pools primary client times with the network included; providers, precision, concurrency, routes and dates differ, retained and new timings can span sessions, and quick failures can reduce all-primary latency. The source published a separate cost ledger on 1 November 2026, stored as decision-reference-v2-cost-per-1000, which supersedes its earlier “cost not consolidated” note. Older 1,342-only scores are not converted into this bank. Not ranked here. Checked / reported: 2026-11-02 · Samples: 40 Snapshot: 2026-11-01 · Revision: f5fa9cfafccd97fcfbdb0336e14284faaa8d2d35 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
System 1 Decision Benchmark v2 · P03 · Tracking current conversational intention	97.500%	Setup & source Tracking current conversational intention: 40 questions. Nimble v4 Max; Hosted native; returned identities {"nimble-v4-max": 1933, "unreported in selected primary": 1}; collected 2026-10-16 to 2026-10-31 America/Toronto; one primary response per input, no scored retries. Unsuccessful responses counted as wrong: {"other provider failure": 1}. Development benchmark, not an independently sampled test: 1,934 questions in 41 categories, the frozen 1,342-question bank plus 192 logical and 400 practical additions. The bank was built with model-guided development, and Jev plus retained Microsoft and Drex feedback are development exposure. Related variants and a single recorded primary response per input limit generalization. The scoring key stays private, so public checks cover coverage, timing and aggregate consistency but cannot rescore answers. Strict accuracy counts a wrong usable answer and an unsuccessful response in the same fixed denominator. Timing pools primary client times with the network included; providers, precision, concurrency, routes and dates differ, retained and new timings can span sessions, and quick failures can reduce all-primary latency. The source published a separate cost ledger on 1 November 2026, stored as decision-reference-v2-cost-per-1000, which supersedes its earlier “cost not consolidated” note. Older 1,342-only scores are not converted into this bank. Not ranked here. Checked / reported: 2026-11-02 · Samples: 40 Snapshot: 2026-11-01 · Revision: f5fa9cfafccd97fcfbdb0336e14284faaa8d2d35 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
System 1 Decision Benchmark v2 · P04 · Selecting a tool or requesting clarification	52.500%	Setup & source Selecting a tool or requesting clarification: 40 questions. Nimble v4 Max; Hosted native; returned identities {"nimble-v4-max": 1933, "unreported in selected primary": 1}; collected 2026-10-16 to 2026-10-31 America/Toronto; one primary response per input, no scored retries. Unsuccessful responses counted as wrong: {"other provider failure": 1}. Development benchmark, not an independently sampled test: 1,934 questions in 41 categories, the frozen 1,342-question bank plus 192 logical and 400 practical additions. The bank was built with model-guided development, and Jev plus retained Microsoft and Drex feedback are development exposure. Related variants and a single recorded primary response per input limit generalization. The scoring key stays private, so public checks cover coverage, timing and aggregate consistency but cannot rescore answers. Strict accuracy counts a wrong usable answer and an unsuccessful response in the same fixed denominator. Timing pools primary client times with the network included; providers, precision, concurrency, routes and dates differ, retained and new timings can span sessions, and quick failures can reduce all-primary latency. The source published a separate cost ledger on 1 November 2026, stored as decision-reference-v2-cost-per-1000, which supersedes its earlier “cost not consolidated” note. Older 1,342-only scores are not converted into this bank. Not ranked here. Checked / reported: 2026-11-02 · Samples: 40 Snapshot: 2026-11-01 · Revision: f5fa9cfafccd97fcfbdb0336e14284faaa8d2d35 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
System 1 Decision Benchmark v2 · P05 · Checking authorization and scope	57.500%	Setup & source Checking authorization and scope: 40 questions. Nimble v4 Max; Hosted native; returned identities {"nimble-v4-max": 1933, "unreported in selected primary": 1}; collected 2026-10-16 to 2026-10-31 America/Toronto; one primary response per input, no scored retries. Unsuccessful responses counted as wrong: {"other provider failure": 1}. Development benchmark, not an independently sampled test: 1,934 questions in 41 categories, the frozen 1,342-question bank plus 192 logical and 400 practical additions. The bank was built with model-guided development, and Jev plus retained Microsoft and Drex feedback are development exposure. Related variants and a single recorded primary response per input limit generalization. The scoring key stays private, so public checks cover coverage, timing and aggregate consistency but cannot rescore answers. Strict accuracy counts a wrong usable answer and an unsuccessful response in the same fixed denominator. Timing pools primary client times with the network included; providers, precision, concurrency, routes and dates differ, retained and new timings can span sessions, and quick failures can reduce all-primary latency. The source published a separate cost ledger on 1 November 2026, stored as decision-reference-v2-cost-per-1000, which supersedes its earlier “cost not consolidated” note. Older 1,342-only scores are not converted into this bank. Not ranked here. Checked / reported: 2026-11-02 · Samples: 40 Snapshot: 2026-11-01 · Revision: f5fa9cfafccd97fcfbdb0336e14284faaa8d2d35 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
System 1 Decision Benchmark v2 · P06 · Making decisions with untrusted content	97.500%	Setup & source Making decisions with untrusted content: 40 questions. Nimble v4 Max; Hosted native; returned identities {"nimble-v4-max": 1933, "unreported in selected primary": 1}; collected 2026-10-16 to 2026-10-31 America/Toronto; one primary response per input, no scored retries. Unsuccessful responses counted as wrong: {"other provider failure": 1}. Development benchmark, not an independently sampled test: 1,934 questions in 41 categories, the frozen 1,342-question bank plus 192 logical and 400 practical additions. The bank was built with model-guided development, and Jev plus retained Microsoft and Drex feedback are development exposure. Related variants and a single recorded primary response per input limit generalization. The scoring key stays private, so public checks cover coverage, timing and aggregate consistency but cannot rescore answers. Strict accuracy counts a wrong usable answer and an unsuccessful response in the same fixed denominator. Timing pools primary client times with the network included; providers, precision, concurrency, routes and dates differ, retained and new timings can span sessions, and quick failures can reduce all-primary latency. The source published a separate cost ledger on 1 November 2026, stored as decision-reference-v2-cost-per-1000, which supersedes its earlier “cost not consolidated” note. Older 1,342-only scores are not converted into this bank. Not ranked here. Checked / reported: 2026-11-02 · Samples: 40 Snapshot: 2026-11-01 · Revision: f5fa9cfafccd97fcfbdb0336e14284faaa8d2d35 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
System 1 Decision Benchmark v2 · P07 · Applying a contextual content policy	85.000%	Setup & source Applying a contextual content policy: 40 questions. Nimble v4 Max; Hosted native; returned identities {"nimble-v4-max": 1933, "unreported in selected primary": 1}; collected 2026-10-16 to 2026-10-31 America/Toronto; one primary response per input, no scored retries. Unsuccessful responses counted as wrong: {"other provider failure": 1}. Development benchmark, not an independently sampled test: 1,934 questions in 41 categories, the frozen 1,342-question bank plus 192 logical and 400 practical additions. The bank was built with model-guided development, and Jev plus retained Microsoft and Drex feedback are development exposure. Related variants and a single recorded primary response per input limit generalization. The scoring key stays private, so public checks cover coverage, timing and aggregate consistency but cannot rescore answers. Strict accuracy counts a wrong usable answer and an unsuccessful response in the same fixed denominator. Timing pools primary client times with the network included; providers, precision, concurrency, routes and dates differ, retained and new timings can span sessions, and quick failures can reduce all-primary latency. The source published a separate cost ledger on 1 November 2026, stored as decision-reference-v2-cost-per-1000, which supersedes its earlier “cost not consolidated” note. Older 1,342-only scores are not converted into this bank. Not ranked here. Checked / reported: 2026-11-02 · Samples: 40 Snapshot: 2026-11-01 · Revision: f5fa9cfafccd97fcfbdb0336e14284faaa8d2d35 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
System 1 Decision Benchmark v2 · P08 · Extracting and normalizing facts	50.000%	Setup & source Extracting and normalizing facts: 40 questions. Nimble v4 Max; Hosted native; returned identities {"nimble-v4-max": 1933, "unreported in selected primary": 1}; collected 2026-10-16 to 2026-10-31 America/Toronto; one primary response per input, no scored retries. Unsuccessful responses counted as wrong: {"other provider failure": 1}. Development benchmark, not an independently sampled test: 1,934 questions in 41 categories, the frozen 1,342-question bank plus 192 logical and 400 practical additions. The bank was built with model-guided development, and Jev plus retained Microsoft and Drex feedback are development exposure. Related variants and a single recorded primary response per input limit generalization. The scoring key stays private, so public checks cover coverage, timing and aggregate consistency but cannot rescore answers. Strict accuracy counts a wrong usable answer and an unsuccessful response in the same fixed denominator. Timing pools primary client times with the network included; providers, precision, concurrency, routes and dates differ, retained and new timings can span sessions, and quick failures can reduce all-primary latency. The source published a separate cost ledger on 1 November 2026, stored as decision-reference-v2-cost-per-1000, which supersedes its earlier “cost not consolidated” note. Older 1,342-only scores are not converted into this bank. Not ranked here. Checked / reported: 2026-11-02 · Samples: 40 Snapshot: 2026-11-01 · Revision: f5fa9cfafccd97fcfbdb0336e14284faaa8d2d35 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
System 1 Decision Benchmark v2 · P09 · Checking claims against supplied evidence	65.000%	Setup & source Checking claims against supplied evidence: 40 questions. Nimble v4 Max; Hosted native; returned identities {"nimble-v4-max": 1933, "unreported in selected primary": 1}; collected 2026-10-16 to 2026-10-31 America/Toronto; one primary response per input, no scored retries. Unsuccessful responses counted as wrong: {"other provider failure": 1}. Development benchmark, not an independently sampled test: 1,934 questions in 41 categories, the frozen 1,342-question bank plus 192 logical and 400 practical additions. The bank was built with model-guided development, and Jev plus retained Microsoft and Drex feedback are development exposure. Related variants and a single recorded primary response per input limit generalization. The scoring key stays private, so public checks cover coverage, timing and aggregate consistency but cannot rescore answers. Strict accuracy counts a wrong usable answer and an unsuccessful response in the same fixed denominator. Timing pools primary client times with the network included; providers, precision, concurrency, routes and dates differ, retained and new timings can span sessions, and quick failures can reduce all-primary latency. The source published a separate cost ledger on 1 November 2026, stored as decision-reference-v2-cost-per-1000, which supersedes its earlier “cost not consolidated” note. Older 1,342-only scores are not converted into this bank. Not ranked here. Checked / reported: 2026-11-02 · Samples: 40 Snapshot: 2026-11-01 · Revision: f5fa9cfafccd97fcfbdb0336e14284faaa8d2d35 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
System 1 Decision Benchmark v2 · P10 · Assigning severity or priority from a rubric	72.500%	Setup & source Assigning severity or priority from a rubric: 40 questions. Nimble v4 Max; Hosted native; returned identities {"nimble-v4-max": 1933, "unreported in selected primary": 1}; collected 2026-10-16 to 2026-10-31 America/Toronto; one primary response per input, no scored retries. Unsuccessful responses counted as wrong: {"other provider failure": 1}. Development benchmark, not an independently sampled test: 1,934 questions in 41 categories, the frozen 1,342-question bank plus 192 logical and 400 practical additions. The bank was built with model-guided development, and Jev plus retained Microsoft and Drex feedback are development exposure. Related variants and a single recorded primary response per input limit generalization. The scoring key stays private, so public checks cover coverage, timing and aggregate consistency but cannot rescore answers. Strict accuracy counts a wrong usable answer and an unsuccessful response in the same fixed denominator. Timing pools primary client times with the network included; providers, precision, concurrency, routes and dates differ, retained and new timings can span sessions, and quick failures can reduce all-primary latency. The source published a separate cost ledger on 1 November 2026, stored as decision-reference-v2-cost-per-1000, which supersedes its earlier “cost not consolidated” note. Older 1,342-only scores are not converted into this bank. Not ranked here. Checked / reported: 2026-11-02 · Samples: 40 Snapshot: 2026-11-01 · Revision: f5fa9cfafccd97fcfbdb0336e14284faaa8d2d35 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
System 1 Decision Benchmark v2 · Tariff-based cost per 1,000 questions	0.02991 USD/1k decisions	Setup & source Captured-tariff estimate for primary calls without a reported charge, in USD per 1,000 questions. Nimble v4 Max; System 1 v2 route; evidence captured_tariff_or_mixed. 1933 calls at a captured tariff, 0 on free access, 1 unknown, out of 1,934. Tariff-based estimate from the same ledger, kept apart from the reported cost dataset. These calls have no provider-reported charge: the ledger applied a captured tariff, and calls on free access during a launch week add nothing, so a value can understate the paid rate. Providers and routes differ; these are not current tariffs. Development benchmark; not ranked here. Checked / reported: 2026-11-02 · Samples: 1934 Snapshot: 2026-11-02 · Revision: 5b47806ba527db5e1abc88578789f3eae6e8e646 · Published summary observations; upstream task/result terms retained Result source · Benchmark source

Evaluator-published · Benchmark Heaven

Nimble-v4-max · Evaluator-published · Benchmark Heaven
Evaluation	Result	Evidence
Benchmark Heaven JevBench v1.8 · Capability Score	81.157	Setup & source Capability Score on the rendered board. Nimble-v4-max (open weights, self-hosted) · Open weights · LLM decoder. Board key nimble-v4-max-open; method tag v1.7. Measured on the Second regular draw (v1.7.0), 2026-10-21; put on the v1.6.1 scale by the draw's anchor offset (see How it works). Source link https://huggingface.co/bespokelabs/Nimble-v4-max. Rendered board row. The same board also has nimble-v4-max at 79.717, kept in published metrics. Benchmark Heaven's rendered board, read 2 November 2026: 283 ranked systems · 1,500 decisions each (600 for 19 equated API re-runs) · 131 newer ranked rows tagged with their draw · 14 older rows dated separately. Newer rows are measured on dated draws and put on the v1.6.1 scale by the draw's anchor offset, so a row's draw is part of its identity. Distinct from Leanmcp JevBench, the community Decision Index and the stored Benchmark Heaven v1.6.1 feed. Cost is the source's model on a GPU-time basis for self-hosted rows, not a provider tariff. Self-hosted latency is timed on the GPU over loopback and API latency end to end with the network; the source says they are not directly comparable, so they stay in separate datasets. No Examenos recomputation. Checked / reported: 2026-11-02 · Evaluated: 2026-10-21 · Samples: Not published Snapshot: 2026-10-31 · Revision: Board v1.7.48, method v1.8, updated 2026-10-31; page sha256 df9093384c9cdf5854e065f71dd3abd83a127793f22f4ca70268e9981e0a0b17 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
Benchmark Heaven JevBench v1.8 · Intelligence	71.0	Setup & source Intelligence on the rendered board. Nimble-v4-max (open weights, self-hosted) · Open weights · LLM decoder. Board key nimble-v4-max-open; method tag v1.7. Measured on the Second regular draw (v1.7.0), 2026-10-21; put on the v1.6.1 scale by the draw's anchor offset (see How it works). Source link https://huggingface.co/bespokelabs/Nimble-v4-max. Rendered board row. The same board also has nimble-v4-max at 79.717, kept in published metrics. Benchmark Heaven's rendered board, read 2 November 2026: 283 ranked systems · 1,500 decisions each (600 for 19 equated API re-runs) · 131 newer ranked rows tagged with their draw · 14 older rows dated separately. Newer rows are measured on dated draws and put on the v1.6.1 scale by the draw's anchor offset, so a row's draw is part of its identity. Distinct from Leanmcp JevBench, the community Decision Index and the stored Benchmark Heaven v1.6.1 feed. Cost is the source's model on a GPU-time basis for self-hosted rows, not a provider tariff. Self-hosted latency is timed on the GPU over loopback and API latency end to end with the network; the source says they are not directly comparable, so they stay in separate datasets. No Examenos recomputation. Checked / reported: 2026-11-02 · Evaluated: 2026-10-21 · Samples: Not published Snapshot: 2026-10-31 · Revision: Board v1.7.48, method v1.8, updated 2026-10-31; page sha256 df9093384c9cdf5854e065f71dd3abd83a127793f22f4ca70268e9981e0a0b17 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
Benchmark Heaven JevBench v1.8 · Calibration	91.4	Setup & source Calibration on the rendered board. Nimble-v4-max (open weights, self-hosted) · Open weights · LLM decoder. Board key nimble-v4-max-open; method tag v1.7. Measured on the Second regular draw (v1.7.0), 2026-10-21; put on the v1.6.1 scale by the draw's anchor offset (see How it works). Source link https://huggingface.co/bespokelabs/Nimble-v4-max. Rendered board row. The same board also has nimble-v4-max at 79.717, kept in published metrics. Benchmark Heaven's rendered board, read 2 November 2026: 283 ranked systems · 1,500 decisions each (600 for 19 equated API re-runs) · 131 newer ranked rows tagged with their draw · 14 older rows dated separately. Newer rows are measured on dated draws and put on the v1.6.1 scale by the draw's anchor offset, so a row's draw is part of its identity. Distinct from Leanmcp JevBench, the community Decision Index and the stored Benchmark Heaven v1.6.1 feed. Cost is the source's model on a GPU-time basis for self-hosted rows, not a provider tariff. Self-hosted latency is timed on the GPU over loopback and API latency end to end with the network; the source says they are not directly comparable, so they stay in separate datasets. No Examenos recomputation. Checked / reported: 2026-11-02 · Evaluated: 2026-10-21 · Samples: Not published Snapshot: 2026-10-31 · Revision: Board v1.7.48, method v1.8, updated 2026-10-31; page sha256 df9093384c9cdf5854e065f71dd3abd83a127793f22f4ca70268e9981e0a0b17 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
Benchmark Heaven JevBench v1.8 · Cost Score	57.5	Setup & source Cost Score on the rendered board. Nimble-v4-max (open weights, self-hosted) · Open weights · LLM decoder. Board key nimble-v4-max-open; method tag v1.7. Measured on the Second regular draw (v1.7.0), 2026-10-21; put on the v1.6.1 scale by the draw's anchor offset (see How it works). Source link https://huggingface.co/bespokelabs/Nimble-v4-max. Rendered board row. The same board also has nimble-v4-max at 79.717, kept in published metrics. Benchmark Heaven's rendered board, read 2 November 2026: 283 ranked systems · 1,500 decisions each (600 for 19 equated API re-runs) · 131 newer ranked rows tagged with their draw · 14 older rows dated separately. Newer rows are measured on dated draws and put on the v1.6.1 scale by the draw's anchor offset, so a row's draw is part of its identity. Distinct from Leanmcp JevBench, the community Decision Index and the stored Benchmark Heaven v1.6.1 feed. Cost is the source's model on a GPU-time basis for self-hosted rows, not a provider tariff. Self-hosted latency is timed on the GPU over loopback and API latency end to end with the network; the source says they are not directly comparable, so they stay in separate datasets. No Examenos recomputation. Checked / reported: 2026-11-02 · Evaluated: 2026-10-21 · Samples: Not published Snapshot: 2026-10-31 · Revision: Board v1.7.48, method v1.8, updated 2026-10-31; page sha256 df9093384c9cdf5854e065f71dd3abd83a127793f22f4ca70268e9981e0a0b17 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
Benchmark Heaven JevBench v1.8 · Modelled cost per 1,000 decisions	0.02615 USD/1k decisions	Setup & source Modelled cost per 1,000 decisions on the rendered board. Nimble-v4-max (open weights, self-hosted) · Open weights · LLM decoder. Board key nimble-v4-max-open; method tag v1.7. Measured on the Second regular draw (v1.7.0), 2026-10-21; put on the v1.6.1 scale by the draw's anchor offset (see How it works). Source link https://huggingface.co/bespokelabs/Nimble-v4-max. Rendered board row. The same board also has nimble-v4-max at 79.717, kept in published metrics. Benchmark Heaven's rendered board, read 2 November 2026: 283 ranked systems · 1,500 decisions each (600 for 19 equated API re-runs) · 131 newer ranked rows tagged with their draw · 14 older rows dated separately. Newer rows are measured on dated draws and put on the v1.6.1 scale by the draw's anchor offset, so a row's draw is part of its identity. Distinct from Leanmcp JevBench, the community Decision Index and the stored Benchmark Heaven v1.6.1 feed. Cost is the source's model on a GPU-time basis for self-hosted rows, not a provider tariff. Self-hosted latency is timed on the GPU over loopback and API latency end to end with the network; the source says they are not directly comparable, so they stay in separate datasets. No Examenos recomputation. Checked / reported: 2026-11-02 · Evaluated: 2026-10-21 · Samples: Not published Snapshot: 2026-10-31 · Revision: Board v1.7.48, method v1.8, updated 2026-10-31; page sha256 df9093384c9cdf5854e065f71dd3abd83a127793f22f4ca70268e9981e0a0b17 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
Benchmark Heaven JevBench v1.8 · Self-hosted p50 on the GPU, loopback	0.154 s	Setup & source Self-hosted p50 on the GPU, loopback on the rendered board. Nimble-v4-max (open weights, self-hosted) · Open weights · LLM decoder. Board key nimble-v4-max-open; method tag v1.7. Measured on the Second regular draw (v1.7.0), 2026-10-21; put on the v1.6.1 scale by the draw's anchor offset (see How it works). Source link https://huggingface.co/bespokelabs/Nimble-v4-max. Rendered board row. The same board also has nimble-v4-max at 79.717, kept in published metrics. Benchmark Heaven's rendered board, read 2 November 2026: 283 ranked systems · 1,500 decisions each (600 for 19 equated API re-runs) · 131 newer ranked rows tagged with their draw · 14 older rows dated separately. Newer rows are measured on dated draws and put on the v1.6.1 scale by the draw's anchor offset, so a row's draw is part of its identity. Distinct from Leanmcp JevBench, the community Decision Index and the stored Benchmark Heaven v1.6.1 feed. Cost is the source's model on a GPU-time basis for self-hosted rows, not a provider tariff. Self-hosted latency is timed on the GPU over loopback and API latency end to end with the network; the source says they are not directly comparable, so they stay in separate datasets. No Examenos recomputation. Checked / reported: 2026-11-02 · Evaluated: 2026-10-21 · Samples: Not published Snapshot: 2026-10-31 · Revision: Board v1.7.48, method v1.8, updated 2026-10-31; page sha256 df9093384c9cdf5854e065f71dd3abd83a127793f22f4ca70268e9981e0a0b17 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
Benchmark Heaven JevBench v1.8 · Self-hosted estimate including network (2x + 150 ms)	0.46 s	Setup & source Self-hosted estimate including network (2x + 150 ms) on the rendered board. Nimble-v4-max (open weights, self-hosted) · Open weights · LLM decoder. Board key nimble-v4-max-open; method tag v1.7. Measured on the Second regular draw (v1.7.0), 2026-10-21; put on the v1.6.1 scale by the draw's anchor offset (see How it works). Source link https://huggingface.co/bespokelabs/Nimble-v4-max. Rendered board row. The same board also has nimble-v4-max at 79.717, kept in published metrics. Benchmark Heaven's rendered board, read 2 November 2026: 283 ranked systems · 1,500 decisions each (600 for 19 equated API re-runs) · 131 newer ranked rows tagged with their draw · 14 older rows dated separately. Newer rows are measured on dated draws and put on the v1.6.1 scale by the draw's anchor offset, so a row's draw is part of its identity. Distinct from Leanmcp JevBench, the community Decision Index and the stored Benchmark Heaven v1.6.1 feed. Cost is the source's model on a GPU-time basis for self-hosted rows, not a provider tariff. Self-hosted latency is timed on the GPU over loopback and API latency end to end with the network; the source says they are not directly comparable, so they stay in separate datasets. No Examenos recomputation. Checked / reported: 2026-11-02 · Evaluated: 2026-10-21 · Samples: Not published Snapshot: 2026-10-31 · Revision: Board v1.7.48, method v1.8, updated 2026-10-31; page sha256 df9093384c9cdf5854e065f71dd3abd83a127793f22f4ca70268e9981e0a0b17 · Published summary observations; upstream task/result terms retained Result source · Benchmark source

Nimble-v4-max · Evaluator-published · Community Decision Index
Evaluation	Result	Evidence
HF text 0.4.1 · Full score	66.56	Setup & source HF text 0.4.1 · Full score Exact source engine nimble-v4-max; {"kind": "full fine-tune", "technique": "autoregressive", "technique_name": "autoregressive", "served_params": 27356728560, "params_estimated": false, "params_from_name": false, "trained_bytes": null, "model_url": "https://huggingface.co/bespokelabs/Nimble-v4-max", "code_url": null, "base_model": "Qwen/Qwen3.8-27B", "base_url": "https://huggingface.co/Qwen/Qwen3.8-27B", "weights_repo": null, "served_checkpoint": null, "org": {"id": "bespokelabs", "name": "Bespoke Labs", "logo": "data/orgs/bespokelabs.png"}, "pr": 226, "new": true}. New in the 2026-10-20 PR batch (PR #226). Submitted public 0.3 run adopted (kit rescore 68.46, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving; decision agreement with the submitted rows 99.85% on 891 rows (bar 98.5%). A separate model from Bespoke Nimble 9B v3, which stays on the board. License CC BY-NC 4.0. Decision Index 0.4.1 (release-v4), source generated 2026-10-29T14:04:16Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Source notes: New in the 2026-10-20 PR batch (PR #226). Submitted public 0.3 run adopted (kit rescore 68.46, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving; decision agreement with the submitted rows 99.85% on 891 rows (bar 98.5%). A separate model from Bespoke Nimble 9B v3, which stays on the board. License CC BY-NC 4.0. Author public-run adoption/execution status and private lab setup are disclosed in configuration; evaluator-published does not mean every public run was independently executed. The board does not name a served checkpoint in this repository history. Community Decision Index 0.4.1: expanded/refreshed harder private new-domain tasks and equating; Full remains 60% private same-skill +40% private new domains, zero public contribution. This population revision is not a model upgrade/regression. Public/native/calibration data remain on their unchanged 0.4 definitions. Source publication/generation is not a model run date. Public submitted runs plus evaluator private measurements are Evaluator-published, not wholly independently executed. Checked / reported: 2026-11-02 · Samples: Not published Snapshot: 2026-10-22T05:40:06Z · Revision: d9962113a1acc95c2d7522c2babfb989afbeb034; c130faf65f82e81d3bdefb01e758c712dc5c5199 for the 2 November 2026 additions · Published observations; result reuse terms not specified Result source · Benchmark source
HF text 0.4 · Public index	68.46	Setup & source HF text 0.4 · Public index Exact source engine nimble-v4-max; {"kind": "full fine-tune", "technique": "autoregressive", "technique_name": "autoregressive", "served_params": 27356728560, "params_estimated": false, "params_from_name": false, "trained_bytes": null, "model_url": "https://huggingface.co/bespokelabs/Nimble-v4-max", "code_url": null, "base_model": "Qwen/Qwen3.8-27B", "base_url": "https://huggingface.co/Qwen/Qwen3.8-27B", "weights_repo": null, "served_checkpoint": null, "org": {"id": "bespokelabs", "name": "Bespoke Labs", "logo": "data/orgs/bespokelabs.png"}, "pr": 226, "new": true}. New in the 2026-10-20 PR batch (PR #226). Submitted public 0.3 run adopted (kit rescore 68.46, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving; decision agreement with the submitted rows 99.85% on 891 rows (bar 98.5%). A separate model from Bespoke Nimble 9B v3, which stays on the board. License CC BY-NC 4.0. Decision Index 0.4.1 (release-v4), source generated 2026-10-29T14:04:16Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Source notes: New in the 2026-10-20 PR batch (PR #226). Submitted public 0.3 run adopted (kit rescore 68.46, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving; decision agreement with the submitted rows 99.85% on 891 rows (bar 98.5%). A separate model from Bespoke Nimble 9B v3, which stays on the board. License CC BY-NC 4.0. Author public-run adoption/execution status and private lab setup are disclosed in configuration; evaluator-published does not mean every public run was independently executed. The board does not name a served checkpoint in this repository history. Decision Index 0.4 (release-v4), source generated 2026-10-20T14:13:56Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Checked / reported: 2026-11-02 · Samples: Not published Snapshot: 2026-10-20T14:13:56Z · Revision: 133313cf8963f2b6bed960abb0152fd14cbb6731; c130faf65f82e81d3bdefb01e758c712dc5c5199 for the 2 November 2026 additions · Published observations; result reuse terms not specified Result source · Benchmark source
HF text 0.4 · Private same-skill component	62.56	Setup & source HF text 0.4 · Private same-skill component Exact source engine nimble-v4-max; {"kind": "full fine-tune", "technique": "autoregressive", "technique_name": "autoregressive", "served_params": 27356728560, "params_estimated": false, "params_from_name": false, "trained_bytes": null, "model_url": "https://huggingface.co/bespokelabs/Nimble-v4-max", "code_url": null, "base_model": "Qwen/Qwen3.8-27B", "base_url": "https://huggingface.co/Qwen/Qwen3.8-27B", "weights_repo": null, "served_checkpoint": null, "org": {"id": "bespokelabs", "name": "Bespoke Labs", "logo": "data/orgs/bespokelabs.png"}, "pr": 226, "new": true}. New in the 2026-10-20 PR batch (PR #226). Submitted public 0.3 run adopted (kit rescore 68.46, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving; decision agreement with the submitted rows 99.85% on 891 rows (bar 98.5%). A separate model from Bespoke Nimble 9B v3, which stays on the board. License CC BY-NC 4.0. Decision Index 0.4.1 (release-v4), source generated 2026-10-29T14:04:16Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Source notes: New in the 2026-10-20 PR batch (PR #226). Submitted public 0.3 run adopted (kit rescore 68.46, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving; decision agreement with the submitted rows 99.85% on 891 rows (bar 98.5%). A separate model from Bespoke Nimble 9B v3, which stays on the board. License CC BY-NC 4.0. Author public-run adoption/execution status and private lab setup are disclosed in configuration; evaluator-published does not mean every public run was independently executed. The board does not name a served checkpoint in this repository history. Decision Index 0.4 (release-v4), source generated 2026-10-20T14:13:56Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Checked / reported: 2026-11-02 · Samples: Not published Snapshot: 2026-10-20T14:13:56Z · Revision: 133313cf8963f2b6bed960abb0152fd14cbb6731; c130faf65f82e81d3bdefb01e758c712dc5c5199 for the 2 November 2026 additions · Published observations; result reuse terms not specified Result source · Benchmark source
HF text 0.4.1 · Private new-domain component	57.01	Setup & source HF text 0.4.1 · Private new-domain component Exact source engine nimble-v4-max; {"kind": "full fine-tune", "technique": "autoregressive", "technique_name": "autoregressive", "served_params": 27356728560, "params_estimated": false, "params_from_name": false, "trained_bytes": null, "model_url": "https://huggingface.co/bespokelabs/Nimble-v4-max", "code_url": null, "base_model": "Qwen/Qwen3.8-27B", "base_url": "https://huggingface.co/Qwen/Qwen3.8-27B", "weights_repo": null, "served_checkpoint": null, "org": {"id": "bespokelabs", "name": "Bespoke Labs", "logo": "data/orgs/bespokelabs.png"}, "pr": 226, "new": true}. New in the 2026-10-20 PR batch (PR #226). Submitted public 0.3 run adopted (kit rescore 68.46, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving; decision agreement with the submitted rows 99.85% on 891 rows (bar 98.5%). A separate model from Bespoke Nimble 9B v3, which stays on the board. License CC BY-NC 4.0. Decision Index 0.4.1 (release-v4), source generated 2026-10-29T14:04:16Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Source notes: New in the 2026-10-20 PR batch (PR #226). Submitted public 0.3 run adopted (kit rescore 68.46, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving; decision agreement with the submitted rows 99.85% on 891 rows (bar 98.5%). A separate model from Bespoke Nimble 9B v3, which stays on the board. License CC BY-NC 4.0. Author public-run adoption/execution status and private lab setup are disclosed in configuration; evaluator-published does not mean every public run was independently executed. The board does not name a served checkpoint in this repository history. Community Decision Index 0.4.1: expanded/refreshed harder private new-domain tasks and equating; Full remains 60% private same-skill +40% private new domains, zero public contribution. This population revision is not a model upgrade/regression. Public/native/calibration data remain on their unchanged 0.4 definitions. Source publication/generation is not a model run date. Public submitted runs plus evaluator private measurements are Evaluator-published, not wholly independently executed. Checked / reported: 2026-11-02 · Samples: Not published Snapshot: 2026-10-22T05:40:06Z · Revision: d9962113a1acc95c2d7522c2babfb989afbeb034; c130faf65f82e81d3bdefb01e758c712dc5c5199 for the 2 November 2026 additions · Published observations; result reuse terms not specified Result source · Benchmark source
HF text 0.4 · Raw public index	75.81	Setup & source HF text 0.4 · Raw public index Exact source engine nimble-v4-max; {"kind": "full fine-tune", "technique": "autoregressive", "technique_name": "autoregressive", "served_params": 27356728560, "params_estimated": false, "params_from_name": false, "trained_bytes": null, "model_url": "https://huggingface.co/bespokelabs/Nimble-v4-max", "code_url": null, "base_model": "Qwen/Qwen3.8-27B", "base_url": "https://huggingface.co/Qwen/Qwen3.8-27B", "weights_repo": null, "served_checkpoint": null, "org": {"id": "bespokelabs", "name": "Bespoke Labs", "logo": "data/orgs/bespokelabs.png"}, "pr": 226, "new": true}. New in the 2026-10-20 PR batch (PR #226). Submitted public 0.3 run adopted (kit rescore 68.46, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving; decision agreement with the submitted rows 99.85% on 891 rows (bar 98.5%). A separate model from Bespoke Nimble 9B v3, which stays on the board. License CC BY-NC 4.0. Decision Index 0.4.1 (release-v4), source generated 2026-10-29T14:04:16Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Source notes: New in the 2026-10-20 PR batch (PR #226). Submitted public 0.3 run adopted (kit rescore 68.46, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving; decision agreement with the submitted rows 99.85% on 891 rows (bar 98.5%). A separate model from Bespoke Nimble 9B v3, which stays on the board. License CC BY-NC 4.0. Author public-run adoption/execution status and private lab setup are disclosed in configuration; evaluator-published does not mean every public run was independently executed. The board does not name a served checkpoint in this repository history. Decision Index 0.4 (release-v4), source generated 2026-10-20T14:13:56Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Checked / reported: 2026-11-02 · Samples: Not published Snapshot: 2026-10-20T14:13:56Z · Revision: 133313cf8963f2b6bed960abb0152fd14cbb6731; c130faf65f82e81d3bdefb01e758c712dc5c5199 for the 2 November 2026 additions · Published observations; result reuse terms not specified Result source · Benchmark source
HF text 0.4 · Public breadth index	67.59	Setup & source HF text 0.4 · Public breadth index Exact source engine nimble-v4-max; {"kind": "full fine-tune", "technique": "autoregressive", "technique_name": "autoregressive", "served_params": 27356728560, "params_estimated": false, "params_from_name": false, "trained_bytes": null, "model_url": "https://huggingface.co/bespokelabs/Nimble-v4-max", "code_url": null, "base_model": "Qwen/Qwen3.8-27B", "base_url": "https://huggingface.co/Qwen/Qwen3.8-27B", "weights_repo": null, "served_checkpoint": null, "org": {"id": "bespokelabs", "name": "Bespoke Labs", "logo": "data/orgs/bespokelabs.png"}, "pr": 226, "new": true}. New in the 2026-10-20 PR batch (PR #226). Submitted public 0.3 run adopted (kit rescore 68.46, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving; decision agreement with the submitted rows 99.85% on 891 rows (bar 98.5%). A separate model from Bespoke Nimble 9B v3, which stays on the board. License CC BY-NC 4.0. Decision Index 0.4.1 (release-v4), source generated 2026-10-29T14:04:16Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Source notes: New in the 2026-10-20 PR batch (PR #226). Submitted public 0.3 run adopted (kit rescore 68.46, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving; decision agreement with the submitted rows 99.85% on 891 rows (bar 98.5%). A separate model from Bespoke Nimble 9B v3, which stays on the board. License CC BY-NC 4.0. Author public-run adoption/execution status and private lab setup are disclosed in configuration; evaluator-published does not mean every public run was independently executed. The board does not name a served checkpoint in this repository history. Decision Index 0.4 (release-v4), source generated 2026-10-20T14:13:56Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Checked / reported: 2026-11-02 · Samples: Not published Snapshot: 2026-10-20T14:13:56Z · Revision: 133313cf8963f2b6bed960abb0152fd14cbb6731; c130faf65f82e81d3bdefb01e758c712dc5c5199 for the 2 November 2026 additions · Published observations; result reuse terms not specified Result source · Benchmark source
HF text 0.4 · Equated private same-skill	62.24	Setup & source HF text 0.4 · Equated private same-skill Exact source engine nimble-v4-max; {"kind": "full fine-tune", "technique": "autoregressive", "technique_name": "autoregressive", "served_params": 27356728560, "params_estimated": false, "params_from_name": false, "trained_bytes": null, "model_url": "https://huggingface.co/bespokelabs/Nimble-v4-max", "code_url": null, "base_model": "Qwen/Qwen3.8-27B", "base_url": "https://huggingface.co/Qwen/Qwen3.8-27B", "weights_repo": null, "served_checkpoint": null, "org": {"id": "bespokelabs", "name": "Bespoke Labs", "logo": "data/orgs/bespokelabs.png"}, "pr": 226, "new": true}. New in the 2026-10-20 PR batch (PR #226). Submitted public 0.3 run adopted (kit rescore 68.46, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving; decision agreement with the submitted rows 99.85% on 891 rows (bar 98.5%). A separate model from Bespoke Nimble 9B v3, which stays on the board. License CC BY-NC 4.0. Decision Index 0.4.1 (release-v4), source generated 2026-10-29T14:04:16Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Source notes: New in the 2026-10-20 PR batch (PR #226). Submitted public 0.3 run adopted (kit rescore 68.46, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving; decision agreement with the submitted rows 99.85% on 891 rows (bar 98.5%). A separate model from Bespoke Nimble 9B v3, which stays on the board. License CC BY-NC 4.0. Author public-run adoption/execution status and private lab setup are disclosed in configuration; evaluator-published does not mean every public run was independently executed. The board does not name a served checkpoint in this repository history. Decision Index 0.4 (release-v4), source generated 2026-10-20T14:13:56Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Checked / reported: 2026-11-02 · Samples: Not published Snapshot: 2026-10-20T14:13:56Z · Revision: 133313cf8963f2b6bed960abb0152fd14cbb6731; c130faf65f82e81d3bdefb01e758c712dc5c5199 for the 2 November 2026 additions · Published observations; result reuse terms not specified Result source · Benchmark source
HF text 0.4.1 · Equated private new domains	73.05	Setup & source HF text 0.4.1 · Equated private new domains Exact source engine nimble-v4-max; {"kind": "full fine-tune", "technique": "autoregressive", "technique_name": "autoregressive", "served_params": 27356728560, "params_estimated": false, "params_from_name": false, "trained_bytes": null, "model_url": "https://huggingface.co/bespokelabs/Nimble-v4-max", "code_url": null, "base_model": "Qwen/Qwen3.8-27B", "base_url": "https://huggingface.co/Qwen/Qwen3.8-27B", "weights_repo": null, "served_checkpoint": null, "org": {"id": "bespokelabs", "name": "Bespoke Labs", "logo": "data/orgs/bespokelabs.png"}, "pr": 226, "new": true}. New in the 2026-10-20 PR batch (PR #226). Submitted public 0.3 run adopted (kit rescore 68.46, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving; decision agreement with the submitted rows 99.85% on 891 rows (bar 98.5%). A separate model from Bespoke Nimble 9B v3, which stays on the board. License CC BY-NC 4.0. Decision Index 0.4.1 (release-v4), source generated 2026-10-29T14:04:16Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Source notes: New in the 2026-10-20 PR batch (PR #226). Submitted public 0.3 run adopted (kit rescore 68.46, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving; decision agreement with the submitted rows 99.85% on 891 rows (bar 98.5%). A separate model from Bespoke Nimble 9B v3, which stays on the board. License CC BY-NC 4.0. Author public-run adoption/execution status and private lab setup are disclosed in configuration; evaluator-published does not mean every public run was independently executed. The board does not name a served checkpoint in this repository history. Community Decision Index 0.4.1: expanded/refreshed harder private new-domain tasks and equating; Full remains 60% private same-skill +40% private new domains, zero public contribution. This population revision is not a model upgrade/regression. Public/native/calibration data remain on their unchanged 0.4 definitions. Source publication/generation is not a model run date. Public submitted runs plus evaluator private measurements are Evaluator-published, not wholly independently executed. Checked / reported: 2026-11-02 · Samples: Not published Snapshot: 2026-10-22T05:40:06Z · Revision: d9962113a1acc95c2d7522c2babfb989afbeb034; c130faf65f82e81d3bdefb01e758c712dc5c5199 for the 2 November 2026 additions · Published observations; result reuse terms not specified Result source · Benchmark source
HF text 0.4 · Full contribution · public	0.00	Setup & source HF text 0.4 · Full contribution · public Exact source engine nimble-v4-max; {"kind": "full fine-tune", "technique": "autoregressive", "technique_name": "autoregressive", "served_params": 27356728560, "params_estimated": false, "params_from_name": false, "trained_bytes": null, "model_url": "https://huggingface.co/bespokelabs/Nimble-v4-max", "code_url": null, "base_model": "Qwen/Qwen3.8-27B", "base_url": "https://huggingface.co/Qwen/Qwen3.8-27B", "weights_repo": null, "served_checkpoint": null, "org": {"id": "bespokelabs", "name": "Bespoke Labs", "logo": "data/orgs/bespokelabs.png"}, "pr": 226, "new": true}. New in the 2026-10-20 PR batch (PR #226). Submitted public 0.3 run adopted (kit rescore 68.46, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving; decision agreement with the submitted rows 99.85% on 891 rows (bar 98.5%). A separate model from Bespoke Nimble 9B v3, which stays on the board. License CC BY-NC 4.0. Decision Index 0.4.1 (release-v4), source generated 2026-10-29T14:04:16Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Source notes: New in the 2026-10-20 PR batch (PR #226). Submitted public 0.3 run adopted (kit rescore 68.46, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving; decision agreement with the submitted rows 99.85% on 891 rows (bar 98.5%). A separate model from Bespoke Nimble 9B v3, which stays on the board. License CC BY-NC 4.0. Author public-run adoption/execution status and private lab setup are disclosed in configuration; evaluator-published does not mean every public run was independently executed. Published weighted contribution; not a distinct benchmark execution. The board does not name a served checkpoint in this repository history. Decision Index 0.4 (release-v4), source generated 2026-10-20T14:13:56Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Checked / reported: 2026-11-02 · Samples: Not published Snapshot: 2026-10-20T14:13:56Z · Revision: 133313cf8963f2b6bed960abb0152fd14cbb6731; c130faf65f82e81d3bdefb01e758c712dc5c5199 for the 2 November 2026 additions · Published observations; result reuse terms not specified Result source · Benchmark source
HF text 0.4 · Full contribution · same_skills	37.34	Setup & source HF text 0.4 · Full contribution · same_skills Exact source engine nimble-v4-max; {"kind": "full fine-tune", "technique": "autoregressive", "technique_name": "autoregressive", "served_params": 27356728560, "params_estimated": false, "params_from_name": false, "trained_bytes": null, "model_url": "https://huggingface.co/bespokelabs/Nimble-v4-max", "code_url": null, "base_model": "Qwen/Qwen3.8-27B", "base_url": "https://huggingface.co/Qwen/Qwen3.8-27B", "weights_repo": null, "served_checkpoint": null, "org": {"id": "bespokelabs", "name": "Bespoke Labs", "logo": "data/orgs/bespokelabs.png"}, "pr": 226, "new": true}. New in the 2026-10-20 PR batch (PR #226). Submitted public 0.3 run adopted (kit rescore 68.46, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving; decision agreement with the submitted rows 99.85% on 891 rows (bar 98.5%). A separate model from Bespoke Nimble 9B v3, which stays on the board. License CC BY-NC 4.0. Decision Index 0.4.1 (release-v4), source generated 2026-10-29T14:04:16Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Source notes: New in the 2026-10-20 PR batch (PR #226). Submitted public 0.3 run adopted (kit rescore 68.46, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving; decision agreement with the submitted rows 99.85% on 891 rows (bar 98.5%). A separate model from Bespoke Nimble 9B v3, which stays on the board. License CC BY-NC 4.0. Author public-run adoption/execution status and private lab setup are disclosed in configuration; evaluator-published does not mean every public run was independently executed. Published weighted contribution; not a distinct benchmark execution. The board does not name a served checkpoint in this repository history. Decision Index 0.4 (release-v4), source generated 2026-10-20T14:13:56Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Checked / reported: 2026-11-02 · Samples: Not published Snapshot: 2026-10-20T14:13:56Z · Revision: 133313cf8963f2b6bed960abb0152fd14cbb6731; c130faf65f82e81d3bdefb01e758c712dc5c5199 for the 2 November 2026 additions · Published observations; result reuse terms not specified Result source · Benchmark source
HF text 0.4.1 · Full contribution · new_domains	29.22	Setup & source HF text 0.4.1 · Full contribution · new_domains Exact source engine nimble-v4-max; {"kind": "full fine-tune", "technique": "autoregressive", "technique_name": "autoregressive", "served_params": 27356728560, "params_estimated": false, "params_from_name": false, "trained_bytes": null, "model_url": "https://huggingface.co/bespokelabs/Nimble-v4-max", "code_url": null, "base_model": "Qwen/Qwen3.8-27B", "base_url": "https://huggingface.co/Qwen/Qwen3.8-27B", "weights_repo": null, "served_checkpoint": null, "org": {"id": "bespokelabs", "name": "Bespoke Labs", "logo": "data/orgs/bespokelabs.png"}, "pr": 226, "new": true}. New in the 2026-10-20 PR batch (PR #226). Submitted public 0.3 run adopted (kit rescore 68.46, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving; decision agreement with the submitted rows 99.85% on 891 rows (bar 98.5%). A separate model from Bespoke Nimble 9B v3, which stays on the board. License CC BY-NC 4.0. Decision Index 0.4.1 (release-v4), source generated 2026-10-29T14:04:16Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Source notes: New in the 2026-10-20 PR batch (PR #226). Submitted public 0.3 run adopted (kit rescore 68.46, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving; decision agreement with the submitted rows 99.85% on 891 rows (bar 98.5%). A separate model from Bespoke Nimble 9B v3, which stays on the board. License CC BY-NC 4.0. Author public-run adoption/execution status and private lab setup are disclosed in configuration; evaluator-published does not mean every public run was independently executed. Published weighted contribution; not a distinct benchmark execution. The board does not name a served checkpoint in this repository history. Community Decision Index 0.4.1: expanded/refreshed harder private new-domain tasks and equating; Full remains 60% private same-skill +40% private new domains, zero public contribution. This population revision is not a model upgrade/regression. Public/native/calibration data remain on their unchanged 0.4 definitions. Source publication/generation is not a model run date. Public submitted runs plus evaluator private measurements are Evaluator-published, not wholly independently executed. Checked / reported: 2026-11-02 · Samples: Not published Snapshot: 2026-10-22T05:40:06Z · Revision: d9962113a1acc95c2d7522c2babfb989afbeb034; c130faf65f82e81d3bdefb01e758c712dc5c5199 for the 2 November 2026 additions · Published observations; result reuse terms not specified Result source · Benchmark source
HF text 0.4 · Knowledge · raw	63.44	Setup & source HF text 0.4 · Knowledge · raw Exact source engine nimble-v4-max; {"kind": "full fine-tune", "technique": "autoregressive", "technique_name": "autoregressive", "served_params": 27356728560, "params_estimated": false, "params_from_name": false, "trained_bytes": null, "model_url": "https://huggingface.co/bespokelabs/Nimble-v4-max", "code_url": null, "base_model": "Qwen/Qwen3.8-27B", "base_url": "https://huggingface.co/Qwen/Qwen3.8-27B", "weights_repo": null, "served_checkpoint": null, "org": {"id": "bespokelabs", "name": "Bespoke Labs", "logo": "data/orgs/bespokelabs.png"}, "pr": 226, "new": true}. New in the 2026-10-20 PR batch (PR #226). Submitted public 0.3 run adopted (kit rescore 68.46, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving; decision agreement with the submitted rows 99.85% on 891 rows (bar 98.5%). A separate model from Bespoke Nimble 9B v3, which stays on the board. License CC BY-NC 4.0. Decision Index 0.4.1 (release-v4), source generated 2026-10-29T14:04:16Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Source notes: New in the 2026-10-20 PR batch (PR #226). Submitted public 0.3 run adopted (kit rescore 68.46, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving; decision agreement with the submitted rows 99.85% on 891 rows (bar 98.5%). A separate model from Bespoke Nimble 9B v3, which stays on the board. License CC BY-NC 4.0. Author public-run adoption/execution status and private lab setup are disclosed in configuration; evaluator-published does not mean every public run was independently executed. Count is contributing benchmarks, not requests. Source area context: {"id": "knowledge", "raw": 0.6344, "skill": 0.5536, "combined": 0.4618, "coverage": 1, "pending": 0, "n": 11} The board does not name a served checkpoint in this repository history. Decision Index 0.4 (release-v4), source generated 2026-10-20T14:13:56Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Checked / reported: 2026-11-02 · Samples: 11 Snapshot: 2026-10-20T14:13:56Z · Revision: 133313cf8963f2b6bed960abb0152fd14cbb6731; c130faf65f82e81d3bdefb01e758c712dc5c5199 for the 2 November 2026 additions · Published observations; result reuse terms not specified Result source · Benchmark source
HF text 0.4 · Knowledge · skill	55.36	Setup & source HF text 0.4 · Knowledge · skill Exact source engine nimble-v4-max; {"kind": "full fine-tune", "technique": "autoregressive", "technique_name": "autoregressive", "served_params": 27356728560, "params_estimated": false, "params_from_name": false, "trained_bytes": null, "model_url": "https://huggingface.co/bespokelabs/Nimble-v4-max", "code_url": null, "base_model": "Qwen/Qwen3.8-27B", "base_url": "https://huggingface.co/Qwen/Qwen3.8-27B", "weights_repo": null, "served_checkpoint": null, "org": {"id": "bespokelabs", "name": "Bespoke Labs", "logo": "data/orgs/bespokelabs.png"}, "pr": 226, "new": true}. New in the 2026-10-20 PR batch (PR #226). Submitted public 0.3 run adopted (kit rescore 68.46, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving; decision agreement with the submitted rows 99.85% on 891 rows (bar 98.5%). A separate model from Bespoke Nimble 9B v3, which stays on the board. License CC BY-NC 4.0. Decision Index 0.4.1 (release-v4), source generated 2026-10-29T14:04:16Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Source notes: New in the 2026-10-20 PR batch (PR #226). Submitted public 0.3 run adopted (kit rescore 68.46, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving; decision agreement with the submitted rows 99.85% on 891 rows (bar 98.5%). A separate model from Bespoke Nimble 9B v3, which stays on the board. License CC BY-NC 4.0. Author public-run adoption/execution status and private lab setup are disclosed in configuration; evaluator-published does not mean every public run was independently executed. Count is contributing benchmarks, not requests. Source area context: {"id": "knowledge", "raw": 0.6344, "skill": 0.5536, "combined": 0.4618, "coverage": 1, "pending": 0, "n": 11} The board does not name a served checkpoint in this repository history. Decision Index 0.4 (release-v4), source generated 2026-10-20T14:13:56Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Checked / reported: 2026-11-02 · Samples: 11 Snapshot: 2026-10-20T14:13:56Z · Revision: 133313cf8963f2b6bed960abb0152fd14cbb6731; c130faf65f82e81d3bdefb01e758c712dc5c5199 for the 2 November 2026 additions · Published observations; result reuse terms not specified Result source · Benchmark source
HF text 0.4 · Knowledge · combined	46.18	Setup & source HF text 0.4 · Knowledge · combined Exact source engine nimble-v4-max; {"kind": "full fine-tune", "technique": "autoregressive", "technique_name": "autoregressive", "served_params": 27356728560, "params_estimated": false, "params_from_name": false, "trained_bytes": null, "model_url": "https://huggingface.co/bespokelabs/Nimble-v4-max", "code_url": null, "base_model": "Qwen/Qwen3.8-27B", "base_url": "https://huggingface.co/Qwen/Qwen3.8-27B", "weights_repo": null, "served_checkpoint": null, "org": {"id": "bespokelabs", "name": "Bespoke Labs", "logo": "data/orgs/bespokelabs.png"}, "pr": 226, "new": true}. New in the 2026-10-20 PR batch (PR #226). Submitted public 0.3 run adopted (kit rescore 68.46, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving; decision agreement with the submitted rows 99.85% on 891 rows (bar 98.5%). A separate model from Bespoke Nimble 9B v3, which stays on the board. License CC BY-NC 4.0. Decision Index 0.4.1 (release-v4), source generated 2026-10-29T14:04:16Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Source notes: New in the 2026-10-20 PR batch (PR #226). Submitted public 0.3 run adopted (kit rescore 68.46, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving; decision agreement with the submitted rows 99.85% on 891 rows (bar 98.5%). A separate model from Bespoke Nimble 9B v3, which stays on the board. License CC BY-NC 4.0. Author public-run adoption/execution status and private lab setup are disclosed in configuration; evaluator-published does not mean every public run was independently executed. Count is contributing benchmarks, not requests. Source area context: {"id": "knowledge", "raw": 0.6344, "skill": 0.5536, "combined": 0.4618, "coverage": 1, "pending": 0, "n": 11} The board does not name a served checkpoint in this repository history. Decision Index 0.4 (release-v4), source generated 2026-10-20T14:13:56Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Checked / reported: 2026-11-02 · Samples: 11 Snapshot: 2026-10-20T14:13:56Z · Revision: 133313cf8963f2b6bed960abb0152fd14cbb6731; c130faf65f82e81d3bdefb01e758c712dc5c5199 for the 2 November 2026 additions · Published observations; result reuse terms not specified Result source · Benchmark source
HF text 0.4 · Language · raw	78.59	Setup & source HF text 0.4 · Language · raw Exact source engine nimble-v4-max; {"kind": "full fine-tune", "technique": "autoregressive", "technique_name": "autoregressive", "served_params": 27356728560, "params_estimated": false, "params_from_name": false, "trained_bytes": null, "model_url": "https://huggingface.co/bespokelabs/Nimble-v4-max", "code_url": null, "base_model": "Qwen/Qwen3.8-27B", "base_url": "https://huggingface.co/Qwen/Qwen3.8-27B", "weights_repo": null, "served_checkpoint": null, "org": {"id": "bespokelabs", "name": "Bespoke Labs", "logo": "data/orgs/bespokelabs.png"}, "pr": 226, "new": true}. New in the 2026-10-20 PR batch (PR #226). Submitted public 0.3 run adopted (kit rescore 68.46, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving; decision agreement with the submitted rows 99.85% on 891 rows (bar 98.5%). A separate model from Bespoke Nimble 9B v3, which stays on the board. License CC BY-NC 4.0. Decision Index 0.4.1 (release-v4), source generated 2026-10-29T14:04:16Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Source notes: New in the 2026-10-20 PR batch (PR #226). Submitted public 0.3 run adopted (kit rescore 68.46, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving; decision agreement with the submitted rows 99.85% on 891 rows (bar 98.5%). A separate model from Bespoke Nimble 9B v3, which stays on the board. License CC BY-NC 4.0. Author public-run adoption/execution status and private lab setup are disclosed in configuration; evaluator-published does not mean every public run was independently executed. Count is contributing benchmarks, not requests. Source area context: {"id": "language", "raw": 0.7859, "skill": 0.7041, "combined": 0.6386, "coverage": 1, "pending": 0, "n": 9} The board does not name a served checkpoint in this repository history. Decision Index 0.4 (release-v4), source generated 2026-10-20T14:13:56Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Checked / reported: 2026-11-02 · Samples: 9 Snapshot: 2026-10-20T14:13:56Z · Revision: 133313cf8963f2b6bed960abb0152fd14cbb6731; c130faf65f82e81d3bdefb01e758c712dc5c5199 for the 2 November 2026 additions · Published observations; result reuse terms not specified Result source · Benchmark source
HF text 0.4 · Language · skill	70.41	Setup & source HF text 0.4 · Language · skill Exact source engine nimble-v4-max; {"kind": "full fine-tune", "technique": "autoregressive", "technique_name": "autoregressive", "served_params": 27356728560, "params_estimated": false, "params_from_name": false, "trained_bytes": null, "model_url": "https://huggingface.co/bespokelabs/Nimble-v4-max", "code_url": null, "base_model": "Qwen/Qwen3.8-27B", "base_url": "https://huggingface.co/Qwen/Qwen3.8-27B", "weights_repo": null, "served_checkpoint": null, "org": {"id": "bespokelabs", "name": "Bespoke Labs", "logo": "data/orgs/bespokelabs.png"}, "pr": 226, "new": true}. New in the 2026-10-20 PR batch (PR #226). Submitted public 0.3 run adopted (kit rescore 68.46, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving; decision agreement with the submitted rows 99.85% on 891 rows (bar 98.5%). A separate model from Bespoke Nimble 9B v3, which stays on the board. License CC BY-NC 4.0. Decision Index 0.4.1 (release-v4), source generated 2026-10-29T14:04:16Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Source notes: New in the 2026-10-20 PR batch (PR #226). Submitted public 0.3 run adopted (kit rescore 68.46, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving; decision agreement with the submitted rows 99.85% on 891 rows (bar 98.5%). A separate model from Bespoke Nimble 9B v3, which stays on the board. License CC BY-NC 4.0. Author public-run adoption/execution status and private lab setup are disclosed in configuration; evaluator-published does not mean every public run was independently executed. Count is contributing benchmarks, not requests. Source area context: {"id": "language", "raw": 0.7859, "skill": 0.7041, "combined": 0.6386, "coverage": 1, "pending": 0, "n": 9} The board does not name a served checkpoint in this repository history. Decision Index 0.4 (release-v4), source generated 2026-10-20T14:13:56Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Checked / reported: 2026-11-02 · Samples: 9 Snapshot: 2026-10-20T14:13:56Z · Revision: 133313cf8963f2b6bed960abb0152fd14cbb6731; c130faf65f82e81d3bdefb01e758c712dc5c5199 for the 2 November 2026 additions · Published observations; result reuse terms not specified Result source · Benchmark source
HF text 0.4 · Language · combined	63.86	Setup & source HF text 0.4 · Language · combined Exact source engine nimble-v4-max; {"kind": "full fine-tune", "technique": "autoregressive", "technique_name": "autoregressive", "served_params": 27356728560, "params_estimated": false, "params_from_name": false, "trained_bytes": null, "model_url": "https://huggingface.co/bespokelabs/Nimble-v4-max", "code_url": null, "base_model": "Qwen/Qwen3.8-27B", "base_url": "https://huggingface.co/Qwen/Qwen3.8-27B", "weights_repo": null, "served_checkpoint": null, "org": {"id": "bespokelabs", "name": "Bespoke Labs", "logo": "data/orgs/bespokelabs.png"}, "pr": 226, "new": true}. New in the 2026-10-20 PR batch (PR #226). Submitted public 0.3 run adopted (kit rescore 68.46, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving; decision agreement with the submitted rows 99.85% on 891 rows (bar 98.5%). A separate model from Bespoke Nimble 9B v3, which stays on the board. License CC BY-NC 4.0. Decision Index 0.4.1 (release-v4), source generated 2026-10-29T14:04:16Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Source notes: New in the 2026-10-20 PR batch (PR #226). Submitted public 0.3 run adopted (kit rescore 68.46, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving; decision agreement with the submitted rows 99.85% on 891 rows (bar 98.5%). A separate model from Bespoke Nimble 9B v3, which stays on the board. License CC BY-NC 4.0. Author public-run adoption/execution status and private lab setup are disclosed in configuration; evaluator-published does not mean every public run was independently executed. Count is contributing benchmarks, not requests. Source area context: {"id": "language", "raw": 0.7859, "skill": 0.7041, "combined": 0.6386, "coverage": 1, "pending": 0, "n": 9} The board does not name a served checkpoint in this repository history. Decision Index 0.4 (release-v4), source generated 2026-10-20T14:13:56Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Checked / reported: 2026-11-02 · Samples: 9 Snapshot: 2026-10-20T14:13:56Z · Revision: 133313cf8963f2b6bed960abb0152fd14cbb6731; c130faf65f82e81d3bdefb01e758c712dc5c5199 for the 2 November 2026 additions · Published observations; result reuse terms not specified Result source · Benchmark source
HF text 0.4 · Retrieval · raw	77.15	Setup & source HF text 0.4 · Retrieval · raw Exact source engine nimble-v4-max; {"kind": "full fine-tune", "technique": "autoregressive", "technique_name": "autoregressive", "served_params": 27356728560, "params_estimated": false, "params_from_name": false, "trained_bytes": null, "model_url": "https://huggingface.co/bespokelabs/Nimble-v4-max", "code_url": null, "base_model": "Qwen/Qwen3.8-27B", "base_url": "https://huggingface.co/Qwen/Qwen3.8-27B", "weights_repo": null, "served_checkpoint": null, "org": {"id": "bespokelabs", "name": "Bespoke Labs", "logo": "data/orgs/bespokelabs.png"}, "pr": 226, "new": true}. New in the 2026-10-20 PR batch (PR #226). Submitted public 0.3 run adopted (kit rescore 68.46, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving; decision agreement with the submitted rows 99.85% on 891 rows (bar 98.5%). A separate model from Bespoke Nimble 9B v3, which stays on the board. License CC BY-NC 4.0. Decision Index 0.4.1 (release-v4), source generated 2026-10-29T14:04:16Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Source notes: New in the 2026-10-20 PR batch (PR #226). Submitted public 0.3 run adopted (kit rescore 68.46, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving; decision agreement with the submitted rows 99.85% on 891 rows (bar 98.5%). A separate model from Bespoke Nimble 9B v3, which stays on the board. License CC BY-NC 4.0. Author public-run adoption/execution status and private lab setup are disclosed in configuration; evaluator-published does not mean every public run was independently executed. Count is contributing benchmarks, not requests. Source area context: {"id": "retrieval", "raw": 0.7715, "skill": 0.7156, "combined": 0.6153, "coverage": 1, "pending": 0, "n": 6} The board does not name a served checkpoint in this repository history. Decision Index 0.4 (release-v4), source generated 2026-10-20T14:13:56Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Checked / reported: 2026-11-02 · Samples: 6 Snapshot: 2026-10-20T14:13:56Z · Revision: 133313cf8963f2b6bed960abb0152fd14cbb6731; c130faf65f82e81d3bdefb01e758c712dc5c5199 for the 2 November 2026 additions · Published observations; result reuse terms not specified Result source · Benchmark source
HF text 0.4 · Retrieval · skill	71.56	Setup & source HF text 0.4 · Retrieval · skill Exact source engine nimble-v4-max; {"kind": "full fine-tune", "technique": "autoregressive", "technique_name": "autoregressive", "served_params": 27356728560, "params_estimated": false, "params_from_name": false, "trained_bytes": null, "model_url": "https://huggingface.co/bespokelabs/Nimble-v4-max", "code_url": null, "base_model": "Qwen/Qwen3.8-27B", "base_url": "https://huggingface.co/Qwen/Qwen3.8-27B", "weights_repo": null, "served_checkpoint": null, "org": {"id": "bespokelabs", "name": "Bespoke Labs", "logo": "data/orgs/bespokelabs.png"}, "pr": 226, "new": true}. New in the 2026-10-20 PR batch (PR #226). Submitted public 0.3 run adopted (kit rescore 68.46, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving; decision agreement with the submitted rows 99.85% on 891 rows (bar 98.5%). A separate model from Bespoke Nimble 9B v3, which stays on the board. License CC BY-NC 4.0. Decision Index 0.4.1 (release-v4), source generated 2026-10-29T14:04:16Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Source notes: New in the 2026-10-20 PR batch (PR #226). Submitted public 0.3 run adopted (kit rescore 68.46, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving; decision agreement with the submitted rows 99.85% on 891 rows (bar 98.5%). A separate model from Bespoke Nimble 9B v3, which stays on the board. License CC BY-NC 4.0. Author public-run adoption/execution status and private lab setup are disclosed in configuration; evaluator-published does not mean every public run was independently executed. Count is contributing benchmarks, not requests. Source area context: {"id": "retrieval", "raw": 0.7715, "skill": 0.7156, "combined": 0.6153, "coverage": 1, "pending": 0, "n": 6} The board does not name a served checkpoint in this repository history. Decision Index 0.4 (release-v4), source generated 2026-10-20T14:13:56Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Checked / reported: 2026-11-02 · Samples: 6 Snapshot: 2026-10-20T14:13:56Z · Revision: 133313cf8963f2b6bed960abb0152fd14cbb6731; c130faf65f82e81d3bdefb01e758c712dc5c5199 for the 2 November 2026 additions · Published observations; result reuse terms not specified Result source · Benchmark source
HF text 0.4 · Retrieval · combined	61.53	Setup & source HF text 0.4 · Retrieval · combined Exact source engine nimble-v4-max; {"kind": "full fine-tune", "technique": "autoregressive", "technique_name": "autoregressive", "served_params": 27356728560, "params_estimated": false, "params_from_name": false, "trained_bytes": null, "model_url": "https://huggingface.co/bespokelabs/Nimble-v4-max", "code_url": null, "base_model": "Qwen/Qwen3.8-27B", "base_url": "https://huggingface.co/Qwen/Qwen3.8-27B", "weights_repo": null, "served_checkpoint": null, "org": {"id": "bespokelabs", "name": "Bespoke Labs", "logo": "data/orgs/bespokelabs.png"}, "pr": 226, "new": true}. New in the 2026-10-20 PR batch (PR #226). Submitted public 0.3 run adopted (kit rescore 68.46, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving; decision agreement with the submitted rows 99.85% on 891 rows (bar 98.5%). A separate model from Bespoke Nimble 9B v3, which stays on the board. License CC BY-NC 4.0. Decision Index 0.4.1 (release-v4), source generated 2026-10-29T14:04:16Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Source notes: New in the 2026-10-20 PR batch (PR #226). Submitted public 0.3 run adopted (kit rescore 68.46, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving; decision agreement with the submitted rows 99.85% on 891 rows (bar 98.5%). A separate model from Bespoke Nimble 9B v3, which stays on the board. License CC BY-NC 4.0. Author public-run adoption/execution status and private lab setup are disclosed in configuration; evaluator-published does not mean every public run was independently executed. Count is contributing benchmarks, not requests. Source area context: {"id": "retrieval", "raw": 0.7715, "skill": 0.7156, "combined": 0.6153, "coverage": 1, "pending": 0, "n": 6} The board does not name a served checkpoint in this repository history. Decision Index 0.4 (release-v4), source generated 2026-10-20T14:13:56Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Checked / reported: 2026-11-02 · Samples: 6 Snapshot: 2026-10-20T14:13:56Z · Revision: 133313cf8963f2b6bed960abb0152fd14cbb6731; c130faf65f82e81d3bdefb01e758c712dc5c5199 for the 2 November 2026 additions · Published observations; result reuse terms not specified Result source · Benchmark source
HF text 0.4 · Tools · raw	90.15	Setup & source HF text 0.4 · Tools · raw Exact source engine nimble-v4-max; {"kind": "full fine-tune", "technique": "autoregressive", "technique_name": "autoregressive", "served_params": 27356728560, "params_estimated": false, "params_from_name": false, "trained_bytes": null, "model_url": "https://huggingface.co/bespokelabs/Nimble-v4-max", "code_url": null, "base_model": "Qwen/Qwen3.8-27B", "base_url": "https://huggingface.co/Qwen/Qwen3.8-27B", "weights_repo": null, "served_checkpoint": null, "org": {"id": "bespokelabs", "name": "Bespoke Labs", "logo": "data/orgs/bespokelabs.png"}, "pr": 226, "new": true}. New in the 2026-10-20 PR batch (PR #226). Submitted public 0.3 run adopted (kit rescore 68.46, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving; decision agreement with the submitted rows 99.85% on 891 rows (bar 98.5%). A separate model from Bespoke Nimble 9B v3, which stays on the board. License CC BY-NC 4.0. Decision Index 0.4.1 (release-v4), source generated 2026-10-29T14:04:16Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Source notes: New in the 2026-10-20 PR batch (PR #226). Submitted public 0.3 run adopted (kit rescore 68.46, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving; decision agreement with the submitted rows 99.85% on 891 rows (bar 98.5%). A separate model from Bespoke Nimble 9B v3, which stays on the board. License CC BY-NC 4.0. Author public-run adoption/execution status and private lab setup are disclosed in configuration; evaluator-published does not mean every public run was independently executed. Count is contributing benchmarks, not requests. Source area context: {"id": "tools", "raw": 0.9015, "skill": 0.8859, "combined": 0.8391, "coverage": 1, "pending": 0, "n": 5} The board does not name a served checkpoint in this repository history. Decision Index 0.4 (release-v4), source generated 2026-10-20T14:13:56Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Checked / reported: 2026-11-02 · Samples: 5 Snapshot: 2026-10-20T14:13:56Z · Revision: 133313cf8963f2b6bed960abb0152fd14cbb6731; c130faf65f82e81d3bdefb01e758c712dc5c5199 for the 2 November 2026 additions · Published observations; result reuse terms not specified Result source · Benchmark source
HF text 0.4 · Tools · skill	88.59	Setup & source HF text 0.4 · Tools · skill Exact source engine nimble-v4-max; {"kind": "full fine-tune", "technique": "autoregressive", "technique_name": "autoregressive", "served_params": 27356728560, "params_estimated": false, "params_from_name": false, "trained_bytes": null, "model_url": "https://huggingface.co/bespokelabs/Nimble-v4-max", "code_url": null, "base_model": "Qwen/Qwen3.8-27B", "base_url": "https://huggingface.co/Qwen/Qwen3.8-27B", "weights_repo": null, "served_checkpoint": null, "org": {"id": "bespokelabs", "name": "Bespoke Labs", "logo": "data/orgs/bespokelabs.png"}, "pr": 226, "new": true}. New in the 2026-10-20 PR batch (PR #226). Submitted public 0.3 run adopted (kit rescore 68.46, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving; decision agreement with the submitted rows 99.85% on 891 rows (bar 98.5%). A separate model from Bespoke Nimble 9B v3, which stays on the board. License CC BY-NC 4.0. Decision Index 0.4.1 (release-v4), source generated 2026-10-29T14:04:16Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Source notes: New in the 2026-10-20 PR batch (PR #226). Submitted public 0.3 run adopted (kit rescore 68.46, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving; decision agreement with the submitted rows 99.85% on 891 rows (bar 98.5%). A separate model from Bespoke Nimble 9B v3, which stays on the board. License CC BY-NC 4.0. Author public-run adoption/execution status and private lab setup are disclosed in configuration; evaluator-published does not mean every public run was independently executed. Count is contributing benchmarks, not requests. Source area context: {"id": "tools", "raw": 0.9015, "skill": 0.8859, "combined": 0.8391, "coverage": 1, "pending": 0, "n": 5} The board does not name a served checkpoint in this repository history. Decision Index 0.4 (release-v4), source generated 2026-10-20T14:13:56Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Checked / reported: 2026-11-02 · Samples: 5 Snapshot: 2026-10-20T14:13:56Z · Revision: 133313cf8963f2b6bed960abb0152fd14cbb6731; c130faf65f82e81d3bdefb01e758c712dc5c5199 for the 2 November 2026 additions · Published observations; result reuse terms not specified Result source · Benchmark source
HF text 0.4 · Tools · combined	83.91	Setup & source HF text 0.4 · Tools · combined Exact source engine nimble-v4-max; {"kind": "full fine-tune", "technique": "autoregressive", "technique_name": "autoregressive", "served_params": 27356728560, "params_estimated": false, "params_from_name": false, "trained_bytes": null, "model_url": "https://huggingface.co/bespokelabs/Nimble-v4-max", "code_url": null, "base_model": "Qwen/Qwen3.8-27B", "base_url": "https://huggingface.co/Qwen/Qwen3.8-27B", "weights_repo": null, "served_checkpoint": null, "org": {"id": "bespokelabs", "name": "Bespoke Labs", "logo": "data/orgs/bespokelabs.png"}, "pr": 226, "new": true}. New in the 2026-10-20 PR batch (PR #226). Submitted public 0.3 run adopted (kit rescore 68.46, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving; decision agreement with the submitted rows 99.85% on 891 rows (bar 98.5%). A separate model from Bespoke Nimble 9B v3, which stays on the board. License CC BY-NC 4.0. Decision Index 0.4.1 (release-v4), source generated 2026-10-29T14:04:16Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Source notes: New in the 2026-10-20 PR batch (PR #226). Submitted public 0.3 run adopted (kit rescore 68.46, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving; decision agreement with the submitted rows 99.85% on 891 rows (bar 98.5%). A separate model from Bespoke Nimble 9B v3, which stays on the board. License CC BY-NC 4.0. Author public-run adoption/execution status and private lab setup are disclosed in configuration; evaluator-published does not mean every public run was independently executed. Count is contributing benchmarks, not requests. Source area context: {"id": "tools", "raw": 0.9015, "skill": 0.8859, "combined": 0.8391, "coverage": 1, "pending": 0, "n": 5} The board does not name a served checkpoint in this repository history. Decision Index 0.4 (release-v4), source generated 2026-10-20T14:13:56Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Checked / reported: 2026-11-02 · Samples: 5 Snapshot: 2026-10-20T14:13:56Z · Revision: 133313cf8963f2b6bed960abb0152fd14cbb6731; c130faf65f82e81d3bdefb01e758c712dc5c5199 for the 2 November 2026 additions · Published observations; result reuse terms not specified Result source · Benchmark source
HF text 0.4 · Arts · raw	71.74	Setup & source HF text 0.4 · Arts · raw Exact source engine nimble-v4-max; {"kind": "full fine-tune", "technique": "autoregressive", "technique_name": "autoregressive", "served_params": 27356728560, "params_estimated": false, "params_from_name": false, "trained_bytes": null, "model_url": "https://huggingface.co/bespokelabs/Nimble-v4-max", "code_url": null, "base_model": "Qwen/Qwen3.8-27B", "base_url": "https://huggingface.co/Qwen/Qwen3.8-27B", "weights_repo": null, "served_checkpoint": null, "org": {"id": "bespokelabs", "name": "Bespoke Labs", "logo": "data/orgs/bespokelabs.png"}, "pr": 226, "new": true}. New in the 2026-10-20 PR batch (PR #226). Submitted public 0.3 run adopted (kit rescore 68.46, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving; decision agreement with the submitted rows 99.85% on 891 rows (bar 98.5%). A separate model from Bespoke Nimble 9B v3, which stays on the board. License CC BY-NC 4.0. Decision Index 0.4.1 (release-v4), source generated 2026-10-29T14:04:16Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Source notes: New in the 2026-10-20 PR batch (PR #226). Submitted public 0.3 run adopted (kit rescore 68.46, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving; decision agreement with the submitted rows 99.85% on 891 rows (bar 98.5%). A separate model from Bespoke Nimble 9B v3, which stays on the board. License CC BY-NC 4.0. Author public-run adoption/execution status and private lab setup are disclosed in configuration; evaluator-published does not mean every public run was independently executed. Count is contributing benchmarks, not requests. Source area context: {"id": "arts", "raw": 0.7174, "skill": 0.5426, "combined": 0.5672, "coverage": 1, "pending": 0, "n": 6} The board does not name a served checkpoint in this repository history. Decision Index 0.4 (release-v4), source generated 2026-10-20T14:13:56Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Checked / reported: 2026-11-02 · Samples: 6 Snapshot: 2026-10-20T14:13:56Z · Revision: 133313cf8963f2b6bed960abb0152fd14cbb6731; c130faf65f82e81d3bdefb01e758c712dc5c5199 for the 2 November 2026 additions · Published observations; result reuse terms not specified Result source · Benchmark source
HF text 0.4 · Arts · skill	54.26	Setup & source HF text 0.4 · Arts · skill Exact source engine nimble-v4-max; {"kind": "full fine-tune", "technique": "autoregressive", "technique_name": "autoregressive", "served_params": 27356728560, "params_estimated": false, "params_from_name": false, "trained_bytes": null, "model_url": "https://huggingface.co/bespokelabs/Nimble-v4-max", "code_url": null, "base_model": "Qwen/Qwen3.8-27B", "base_url": "https://huggingface.co/Qwen/Qwen3.8-27B", "weights_repo": null, "served_checkpoint": null, "org": {"id": "bespokelabs", "name": "Bespoke Labs", "logo": "data/orgs/bespokelabs.png"}, "pr": 226, "new": true}. New in the 2026-10-20 PR batch (PR #226). Submitted public 0.3 run adopted (kit rescore 68.46, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving; decision agreement with the submitted rows 99.85% on 891 rows (bar 98.5%). A separate model from Bespoke Nimble 9B v3, which stays on the board. License CC BY-NC 4.0. Decision Index 0.4.1 (release-v4), source generated 2026-10-29T14:04:16Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Source notes: New in the 2026-10-20 PR batch (PR #226). Submitted public 0.3 run adopted (kit rescore 68.46, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving; decision agreement with the submitted rows 99.85% on 891 rows (bar 98.5%). A separate model from Bespoke Nimble 9B v3, which stays on the board. License CC BY-NC 4.0. Author public-run adoption/execution status and private lab setup are disclosed in configuration; evaluator-published does not mean every public run was independently executed. Count is contributing benchmarks, not requests. Source area context: {"id": "arts", "raw": 0.7174, "skill": 0.5426, "combined": 0.5672, "coverage": 1, "pending": 0, "n": 6} The board does not name a served checkpoint in this repository history. Decision Index 0.4 (release-v4), source generated 2026-10-20T14:13:56Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Checked / reported: 2026-11-02 · Samples: 6 Snapshot: 2026-10-20T14:13:56Z · Revision: 133313cf8963f2b6bed960abb0152fd14cbb6731; c130faf65f82e81d3bdefb01e758c712dc5c5199 for the 2 November 2026 additions · Published observations; result reuse terms not specified Result source · Benchmark source
HF text 0.4 · Arts · combined	56.72	Setup & source HF text 0.4 · Arts · combined Exact source engine nimble-v4-max; {"kind": "full fine-tune", "technique": "autoregressive", "technique_name": "autoregressive", "served_params": 27356728560, "params_estimated": false, "params_from_name": false, "trained_bytes": null, "model_url": "https://huggingface.co/bespokelabs/Nimble-v4-max", "code_url": null, "base_model": "Qwen/Qwen3.8-27B", "base_url": "https://huggingface.co/Qwen/Qwen3.8-27B", "weights_repo": null, "served_checkpoint": null, "org": {"id": "bespokelabs", "name": "Bespoke Labs", "logo": "data/orgs/bespokelabs.png"}, "pr": 226, "new": true}. New in the 2026-10-20 PR batch (PR #226). Submitted public 0.3 run adopted (kit rescore 68.46, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving; decision agreement with the submitted rows 99.85% on 891 rows (bar 98.5%). A separate model from Bespoke Nimble 9B v3, which stays on the board. License CC BY-NC 4.0. Decision Index 0.4.1 (release-v4), source generated 2026-10-29T14:04:16Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Source notes: New in the 2026-10-20 PR batch (PR #226). Submitted public 0.3 run adopted (kit rescore 68.46, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving; decision agreement with the submitted rows 99.85% on 891 rows (bar 98.5%). A separate model from Bespoke Nimble 9B v3, which stays on the board. License CC BY-NC 4.0. Author public-run adoption/execution status and private lab setup are disclosed in configuration; evaluator-published does not mean every public run was independently executed. Count is contributing benchmarks, not requests. Source area context: {"id": "arts", "raw": 0.7174, "skill": 0.5426, "combined": 0.5672, "coverage": 1, "pending": 0, "n": 6} The board does not name a served checkpoint in this repository history. Decision Index 0.4 (release-v4), source generated 2026-10-20T14:13:56Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Checked / reported: 2026-11-02 · Samples: 6 Snapshot: 2026-10-20T14:13:56Z · Revision: 133313cf8963f2b6bed960abb0152fd14cbb6731; c130faf65f82e81d3bdefb01e758c712dc5c5199 for the 2 November 2026 additions · Published observations; result reuse terms not specified Result source · Benchmark source
HF text calibration 0.4 · ECE	6.35 pp	Setup & source HF text calibration 0.4 · ECE Exact source engine nimble-v4-max; {"kind": "full fine-tune", "technique": "autoregressive", "technique_name": "autoregressive", "served_params": 27356728560, "params_estimated": false, "params_from_name": false, "trained_bytes": null, "model_url": "https://huggingface.co/bespokelabs/Nimble-v4-max", "code_url": null, "base_model": "Qwen/Qwen3.8-27B", "base_url": "https://huggingface.co/Qwen/Qwen3.8-27B", "weights_repo": null, "served_checkpoint": null, "org": {"id": "bespokelabs", "name": "Bespoke Labs", "logo": "data/orgs/bespokelabs.png"}, "pr": 226, "new": true}. New in the 2026-10-20 PR batch (PR #226). Submitted public 0.3 run adopted (kit rescore 68.46, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving; decision agreement with the submitted rows 99.85% on 891 rows (bar 98.5%). A separate model from Bespoke Nimble 9B v3, which stays on the board. License CC BY-NC 4.0. Decision Index 0.4.1 (release-v4), source generated 2026-10-29T14:04:16Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Source notes: New in the 2026-10-20 PR batch (PR #226). Submitted public 0.3 run adopted (kit rescore 68.46, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving; decision agreement with the submitted rows 99.85% on 891 rows (bar 98.5%). A separate model from Bespoke Nimble 9B v3, which stays on the board. License CC BY-NC 4.0. Author public-run adoption/execution status and private lab setup are disclosed in configuration; evaluator-published does not mean every public run was independently executed. 32 benchmarks with a per-field right or wrong answer (ToolRet, BRIGHT and RouterBench excluded), each weighted equally; a 1-in-6 sample of requests; confidence is the probability placed on the chosen option. Rows an entrant trained on count as wrong here too. Contributing benchmarks 31. The board does not name a served checkpoint in this repository history. Decision Index 0.4 (release-v4), source generated 2026-10-20T14:13:56Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Checked / reported: 2026-11-02 · Samples: 34406 Snapshot: 2026-10-20T14:13:56Z · Revision: 133313cf8963f2b6bed960abb0152fd14cbb6731; c130faf65f82e81d3bdefb01e758c712dc5c5199 for the 2 November 2026 additions · Published observations; result reuse terms not specified Result source · Benchmark source
HF text calibration 0.4 · Brier	0.2875 Brier	Setup & source HF text calibration 0.4 · Brier Exact source engine nimble-v4-max; {"kind": "full fine-tune", "technique": "autoregressive", "technique_name": "autoregressive", "served_params": 27356728560, "params_estimated": false, "params_from_name": false, "trained_bytes": null, "model_url": "https://huggingface.co/bespokelabs/Nimble-v4-max", "code_url": null, "base_model": "Qwen/Qwen3.8-27B", "base_url": "https://huggingface.co/Qwen/Qwen3.8-27B", "weights_repo": null, "served_checkpoint": null, "org": {"id": "bespokelabs", "name": "Bespoke Labs", "logo": "data/orgs/bespokelabs.png"}, "pr": 226, "new": true}. New in the 2026-10-20 PR batch (PR #226). Submitted public 0.3 run adopted (kit rescore 68.46, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving; decision agreement with the submitted rows 99.85% on 891 rows (bar 98.5%). A separate model from Bespoke Nimble 9B v3, which stays on the board. License CC BY-NC 4.0. Decision Index 0.4.1 (release-v4), source generated 2026-10-29T14:04:16Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Source notes: New in the 2026-10-20 PR batch (PR #226). Submitted public 0.3 run adopted (kit rescore 68.46, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving; decision agreement with the submitted rows 99.85% on 891 rows (bar 98.5%). A separate model from Bespoke Nimble 9B v3, which stays on the board. License CC BY-NC 4.0. Author public-run adoption/execution status and private lab setup are disclosed in configuration; evaluator-published does not mean every public run was independently executed. 32 benchmarks with a per-field right or wrong answer (ToolRet, BRIGHT and RouterBench excluded), each weighted equally; a 1-in-6 sample of requests; confidence is the probability placed on the chosen option. Rows an entrant trained on count as wrong here too. Contributing benchmarks 31. The board does not name a served checkpoint in this repository history. Decision Index 0.4 (release-v4), source generated 2026-10-20T14:13:56Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Checked / reported: 2026-11-02 · Samples: 34406 Snapshot: 2026-10-20T14:13:56Z · Revision: 133313cf8963f2b6bed960abb0152fd14cbb6731; c130faf65f82e81d3bdefb01e758c712dc5c5199 for the 2 November 2026 additions · Published observations; result reuse terms not specified Result source · Benchmark source
HF text calibration 0.4 · Sample accuracy	79.38%	Setup & source HF text calibration 0.4 · Sample accuracy Exact source engine nimble-v4-max; {"kind": "full fine-tune", "technique": "autoregressive", "technique_name": "autoregressive", "served_params": 27356728560, "params_estimated": false, "params_from_name": false, "trained_bytes": null, "model_url": "https://huggingface.co/bespokelabs/Nimble-v4-max", "code_url": null, "base_model": "Qwen/Qwen3.8-27B", "base_url": "https://huggingface.co/Qwen/Qwen3.8-27B", "weights_repo": null, "served_checkpoint": null, "org": {"id": "bespokelabs", "name": "Bespoke Labs", "logo": "data/orgs/bespokelabs.png"}, "pr": 226, "new": true}. New in the 2026-10-20 PR batch (PR #226). Submitted public 0.3 run adopted (kit rescore 68.46, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving; decision agreement with the submitted rows 99.85% on 891 rows (bar 98.5%). A separate model from Bespoke Nimble 9B v3, which stays on the board. License CC BY-NC 4.0. Decision Index 0.4.1 (release-v4), source generated 2026-10-29T14:04:16Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Source notes: New in the 2026-10-20 PR batch (PR #226). Submitted public 0.3 run adopted (kit rescore 68.46, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving; decision agreement with the submitted rows 99.85% on 891 rows (bar 98.5%). A separate model from Bespoke Nimble 9B v3, which stays on the board. License CC BY-NC 4.0. Author public-run adoption/execution status and private lab setup are disclosed in configuration; evaluator-published does not mean every public run was independently executed. 32 benchmarks with a per-field right or wrong answer (ToolRet, BRIGHT and RouterBench excluded), each weighted equally; a 1-in-6 sample of requests; confidence is the probability placed on the chosen option. Rows an entrant trained on count as wrong here too. Contributing benchmarks 31. The board does not name a served checkpoint in this repository history. Decision Index 0.4 (release-v4), source generated 2026-10-20T14:13:56Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Checked / reported: 2026-11-02 · Samples: 34406 Snapshot: 2026-10-20T14:13:56Z · Revision: 133313cf8963f2b6bed960abb0152fd14cbb6731; c130faf65f82e81d3bdefb01e758c712dc5c5199 for the 2 November 2026 additions · Published observations; result reuse terms not specified Result source · Benchmark source
HF text calibration 0.4 · Mean chosen confidence	85.45%	Setup & source HF text calibration 0.4 · Mean chosen confidence Exact source engine nimble-v4-max; {"kind": "full fine-tune", "technique": "autoregressive", "technique_name": "autoregressive", "served_params": 27356728560, "params_estimated": false, "params_from_name": false, "trained_bytes": null, "model_url": "https://huggingface.co/bespokelabs/Nimble-v4-max", "code_url": null, "base_model": "Qwen/Qwen3.8-27B", "base_url": "https://huggingface.co/Qwen/Qwen3.8-27B", "weights_repo": null, "served_checkpoint": null, "org": {"id": "bespokelabs", "name": "Bespoke Labs", "logo": "data/orgs/bespokelabs.png"}, "pr": 226, "new": true}. New in the 2026-10-20 PR batch (PR #226). Submitted public 0.3 run adopted (kit rescore 68.46, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving; decision agreement with the submitted rows 99.85% on 891 rows (bar 98.5%). A separate model from Bespoke Nimble 9B v3, which stays on the board. License CC BY-NC 4.0. Decision Index 0.4.1 (release-v4), source generated 2026-10-29T14:04:16Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Source notes: New in the 2026-10-20 PR batch (PR #226). Submitted public 0.3 run adopted (kit rescore 68.46, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving; decision agreement with the submitted rows 99.85% on 891 rows (bar 98.5%). A separate model from Bespoke Nimble 9B v3, which stays on the board. License CC BY-NC 4.0. Author public-run adoption/execution status and private lab setup are disclosed in configuration; evaluator-published does not mean every public run was independently executed. 32 benchmarks with a per-field right or wrong answer (ToolRet, BRIGHT and RouterBench excluded), each weighted equally; a 1-in-6 sample of requests; confidence is the probability placed on the chosen option. Rows an entrant trained on count as wrong here too. Contributing benchmarks 31. The board does not name a served checkpoint in this repository history. Decision Index 0.4 (release-v4), source generated 2026-10-20T14:13:56Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Checked / reported: 2026-11-02 · Samples: 34406 Snapshot: 2026-10-20T14:13:56Z · Revision: 133313cf8963f2b6bed960abb0152fd14cbb6731; c130faf65f82e81d3bdefb01e758c712dc5c5199 for the 2 November 2026 additions · Published observations; result reuse terms not specified Result source · Benchmark source
HF text calibration 0.4 · Wrong among answers at ≥95% confidence	2.31%	Setup & source HF text calibration 0.4 · Wrong among answers at ≥95% confidence Exact source engine nimble-v4-max; {"kind": "full fine-tune", "technique": "autoregressive", "technique_name": "autoregressive", "served_params": 27356728560, "params_estimated": false, "params_from_name": false, "trained_bytes": null, "model_url": "https://huggingface.co/bespokelabs/Nimble-v4-max", "code_url": null, "base_model": "Qwen/Qwen3.8-27B", "base_url": "https://huggingface.co/Qwen/Qwen3.8-27B", "weights_repo": null, "served_checkpoint": null, "org": {"id": "bespokelabs", "name": "Bespoke Labs", "logo": "data/orgs/bespokelabs.png"}, "pr": 226, "new": true}. New in the 2026-10-20 PR batch (PR #226). Submitted public 0.3 run adopted (kit rescore 68.46, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving; decision agreement with the submitted rows 99.85% on 891 rows (bar 98.5%). A separate model from Bespoke Nimble 9B v3, which stays on the board. License CC BY-NC 4.0. Decision Index 0.4.1 (release-v4), source generated 2026-10-29T14:04:16Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Source notes: New in the 2026-10-20 PR batch (PR #226). Submitted public 0.3 run adopted (kit rescore 68.46, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving; decision agreement with the submitted rows 99.85% on 891 rows (bar 98.5%). A separate model from Bespoke Nimble 9B v3, which stays on the board. License CC BY-NC 4.0. Author public-run adoption/execution status and private lab setup are disclosed in configuration; evaluator-published does not mean every public run was independently executed. 32 benchmarks with a per-field right or wrong answer (ToolRet, BRIGHT and RouterBench excluded), each weighted equally; a 1-in-6 sample of requests; confidence is the probability placed on the chosen option. Rows an entrant trained on count as wrong here too. Contributing benchmarks 31. The board does not name a served checkpoint in this repository history. Decision Index 0.4 (release-v4), source generated 2026-10-20T14:13:56Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Checked / reported: 2026-11-02 · Samples: 34406 Snapshot: 2026-10-20T14:13:56Z · Revision: 133313cf8963f2b6bed960abb0152fd14cbb6731; c130faf65f82e81d3bdefb01e758c712dc5c5199 for the 2 November 2026 additions · Published observations; result reuse terms not specified Result source · Benchmark source
HF text calibration 0.4 · Knowledge · ECE	11.01 pp	Setup & source HF text calibration 0.4 · Knowledge · ECE Exact source engine nimble-v4-max; {"kind": "full fine-tune", "technique": "autoregressive", "technique_name": "autoregressive", "served_params": 27356728560, "params_estimated": false, "params_from_name": false, "trained_bytes": null, "model_url": "https://huggingface.co/bespokelabs/Nimble-v4-max", "code_url": null, "base_model": "Qwen/Qwen3.8-27B", "base_url": "https://huggingface.co/Qwen/Qwen3.8-27B", "weights_repo": null, "served_checkpoint": null, "org": {"id": "bespokelabs", "name": "Bespoke Labs", "logo": "data/orgs/bespokelabs.png"}, "pr": 226, "new": true}. New in the 2026-10-20 PR batch (PR #226). Submitted public 0.3 run adopted (kit rescore 68.46, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving; decision agreement with the submitted rows 99.85% on 891 rows (bar 98.5%). A separate model from Bespoke Nimble 9B v3, which stays on the board. License CC BY-NC 4.0. Decision Index 0.4.1 (release-v4), source generated 2026-10-29T14:04:16Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Source notes: New in the 2026-10-20 PR batch (PR #226). Submitted public 0.3 run adopted (kit rescore 68.46, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving; decision agreement with the submitted rows 99.85% on 891 rows (bar 98.5%). A separate model from Bespoke Nimble 9B v3, which stays on the board. License CC BY-NC 4.0. Author public-run adoption/execution status and private lab setup are disclosed in configuration; evaluator-published does not mean every public run was independently executed. Area-specific item counts unpublished; do not use whole-sample count. The board does not name a served checkpoint in this repository history. Decision Index 0.4 (release-v4), source generated 2026-10-20T14:13:56Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Checked / reported: 2026-11-02 · Samples: Not published Snapshot: 2026-10-20T14:13:56Z · Revision: 133313cf8963f2b6bed960abb0152fd14cbb6731; c130faf65f82e81d3bdefb01e758c712dc5c5199 for the 2 November 2026 additions · Published observations; result reuse terms not specified Result source · Benchmark source
HF text calibration 0.4 · Knowledge · Sample accuracy	70.59%	Setup & source HF text calibration 0.4 · Knowledge · Sample accuracy Exact source engine nimble-v4-max; {"kind": "full fine-tune", "technique": "autoregressive", "technique_name": "autoregressive", "served_params": 27356728560, "params_estimated": false, "params_from_name": false, "trained_bytes": null, "model_url": "https://huggingface.co/bespokelabs/Nimble-v4-max", "code_url": null, "base_model": "Qwen/Qwen3.8-27B", "base_url": "https://huggingface.co/Qwen/Qwen3.8-27B", "weights_repo": null, "served_checkpoint": null, "org": {"id": "bespokelabs", "name": "Bespoke Labs", "logo": "data/orgs/bespokelabs.png"}, "pr": 226, "new": true}. New in the 2026-10-20 PR batch (PR #226). Submitted public 0.3 run adopted (kit rescore 68.46, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving; decision agreement with the submitted rows 99.85% on 891 rows (bar 98.5%). A separate model from Bespoke Nimble 9B v3, which stays on the board. License CC BY-NC 4.0. Decision Index 0.4.1 (release-v4), source generated 2026-10-29T14:04:16Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Source notes: New in the 2026-10-20 PR batch (PR #226). Submitted public 0.3 run adopted (kit rescore 68.46, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving; decision agreement with the submitted rows 99.85% on 891 rows (bar 98.5%). A separate model from Bespoke Nimble 9B v3, which stays on the board. License CC BY-NC 4.0. Author public-run adoption/execution status and private lab setup are disclosed in configuration; evaluator-published does not mean every public run was independently executed. Area-specific item counts unpublished; do not use whole-sample count. The board does not name a served checkpoint in this repository history. Decision Index 0.4 (release-v4), source generated 2026-10-20T14:13:56Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Checked / reported: 2026-11-02 · Samples: Not published Snapshot: 2026-10-20T14:13:56Z · Revision: 133313cf8963f2b6bed960abb0152fd14cbb6731; c130faf65f82e81d3bdefb01e758c712dc5c5199 for the 2 November 2026 additions · Published observations; result reuse terms not specified Result source · Benchmark source
HF text calibration 0.4 · Knowledge · Mean chosen confidence	80.46%	Setup & source HF text calibration 0.4 · Knowledge · Mean chosen confidence Exact source engine nimble-v4-max; {"kind": "full fine-tune", "technique": "autoregressive", "technique_name": "autoregressive", "served_params": 27356728560, "params_estimated": false, "params_from_name": false, "trained_bytes": null, "model_url": "https://huggingface.co/bespokelabs/Nimble-v4-max", "code_url": null, "base_model": "Qwen/Qwen3.8-27B", "base_url": "https://huggingface.co/Qwen/Qwen3.8-27B", "weights_repo": null, "served_checkpoint": null, "org": {"id": "bespokelabs", "name": "Bespoke Labs", "logo": "data/orgs/bespokelabs.png"}, "pr": 226, "new": true}. New in the 2026-10-20 PR batch (PR #226). Submitted public 0.3 run adopted (kit rescore 68.46, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving; decision agreement with the submitted rows 99.85% on 891 rows (bar 98.5%). A separate model from Bespoke Nimble 9B v3, which stays on the board. License CC BY-NC 4.0. Decision Index 0.4.1 (release-v4), source generated 2026-10-29T14:04:16Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Source notes: New in the 2026-10-20 PR batch (PR #226). Submitted public 0.3 run adopted (kit rescore 68.46, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving; decision agreement with the submitted rows 99.85% on 891 rows (bar 98.5%). A separate model from Bespoke Nimble 9B v3, which stays on the board. License CC BY-NC 4.0. Author public-run adoption/execution status and private lab setup are disclosed in configuration; evaluator-published does not mean every public run was independently executed. Area-specific item counts unpublished; do not use whole-sample count. The board does not name a served checkpoint in this repository history. Decision Index 0.4 (release-v4), source generated 2026-10-20T14:13:56Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Checked / reported: 2026-11-02 · Samples: Not published Snapshot: 2026-10-20T14:13:56Z · Revision: 133313cf8963f2b6bed960abb0152fd14cbb6731; c130faf65f82e81d3bdefb01e758c712dc5c5199 for the 2 November 2026 additions · Published observations; result reuse terms not specified Result source · Benchmark source
HF text calibration 0.4 · Language · ECE	1.48 pp	Setup & source HF text calibration 0.4 · Language · ECE Exact source engine nimble-v4-max; {"kind": "full fine-tune", "technique": "autoregressive", "technique_name": "autoregressive", "served_params": 27356728560, "params_estimated": false, "params_from_name": false, "trained_bytes": null, "model_url": "https://huggingface.co/bespokelabs/Nimble-v4-max", "code_url": null, "base_model": "Qwen/Qwen3.8-27B", "base_url": "https://huggingface.co/Qwen/Qwen3.8-27B", "weights_repo": null, "served_checkpoint": null, "org": {"id": "bespokelabs", "name": "Bespoke Labs", "logo": "data/orgs/bespokelabs.png"}, "pr": 226, "new": true}. New in the 2026-10-20 PR batch (PR #226). Submitted public 0.3 run adopted (kit rescore 68.46, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving; decision agreement with the submitted rows 99.85% on 891 rows (bar 98.5%). A separate model from Bespoke Nimble 9B v3, which stays on the board. License CC BY-NC 4.0. Decision Index 0.4.1 (release-v4), source generated 2026-10-29T14:04:16Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Source notes: New in the 2026-10-20 PR batch (PR #226). Submitted public 0.3 run adopted (kit rescore 68.46, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving; decision agreement with the submitted rows 99.85% on 891 rows (bar 98.5%). A separate model from Bespoke Nimble 9B v3, which stays on the board. License CC BY-NC 4.0. Author public-run adoption/execution status and private lab setup are disclosed in configuration; evaluator-published does not mean every public run was independently executed. Area-specific item counts unpublished; do not use whole-sample count. The board does not name a served checkpoint in this repository history. Decision Index 0.4 (release-v4), source generated 2026-10-20T14:13:56Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Checked / reported: 2026-11-02 · Samples: Not published Snapshot: 2026-10-20T14:13:56Z · Revision: 133313cf8963f2b6bed960abb0152fd14cbb6731; c130faf65f82e81d3bdefb01e758c712dc5c5199 for the 2 November 2026 additions · Published observations; result reuse terms not specified Result source · Benchmark source
HF text calibration 0.4 · Language · Sample accuracy	89.35%	Setup & source HF text calibration 0.4 · Language · Sample accuracy Exact source engine nimble-v4-max; {"kind": "full fine-tune", "technique": "autoregressive", "technique_name": "autoregressive", "served_params": 27356728560, "params_estimated": false, "params_from_name": false, "trained_bytes": null, "model_url": "https://huggingface.co/bespokelabs/Nimble-v4-max", "code_url": null, "base_model": "Qwen/Qwen3.8-27B", "base_url": "https://huggingface.co/Qwen/Qwen3.8-27B", "weights_repo": null, "served_checkpoint": null, "org": {"id": "bespokelabs", "name": "Bespoke Labs", "logo": "data/orgs/bespokelabs.png"}, "pr": 226, "new": true}. New in the 2026-10-20 PR batch (PR #226). Submitted public 0.3 run adopted (kit rescore 68.46, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving; decision agreement with the submitted rows 99.85% on 891 rows (bar 98.5%). A separate model from Bespoke Nimble 9B v3, which stays on the board. License CC BY-NC 4.0. Decision Index 0.4.1 (release-v4), source generated 2026-10-29T14:04:16Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Source notes: New in the 2026-10-20 PR batch (PR #226). Submitted public 0.3 run adopted (kit rescore 68.46, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving; decision agreement with the submitted rows 99.85% on 891 rows (bar 98.5%). A separate model from Bespoke Nimble 9B v3, which stays on the board. License CC BY-NC 4.0. Author public-run adoption/execution status and private lab setup are disclosed in configuration; evaluator-published does not mean every public run was independently executed. Area-specific item counts unpublished; do not use whole-sample count. The board does not name a served checkpoint in this repository history. Decision Index 0.4 (release-v4), source generated 2026-10-20T14:13:56Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Checked / reported: 2026-11-02 · Samples: Not published Snapshot: 2026-10-20T14:13:56Z · Revision: 133313cf8963f2b6bed960abb0152fd14cbb6731; c130faf65f82e81d3bdefb01e758c712dc5c5199 for the 2 November 2026 additions · Published observations; result reuse terms not specified Result source · Benchmark source
HF text calibration 0.4 · Language · Mean chosen confidence	90.83%	Setup & source HF text calibration 0.4 · Language · Mean chosen confidence Exact source engine nimble-v4-max; {"kind": "full fine-tune", "technique": "autoregressive", "technique_name": "autoregressive", "served_params": 27356728560, "params_estimated": false, "params_from_name": false, "trained_bytes": null, "model_url": "https://huggingface.co/bespokelabs/Nimble-v4-max", "code_url": null, "base_model": "Qwen/Qwen3.8-27B", "base_url": "https://huggingface.co/Qwen/Qwen3.8-27B", "weights_repo": null, "served_checkpoint": null, "org": {"id": "bespokelabs", "name": "Bespoke Labs", "logo": "data/orgs/bespokelabs.png"}, "pr": 226, "new": true}. New in the 2026-10-20 PR batch (PR #226). Submitted public 0.3 run adopted (kit rescore 68.46, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving; decision agreement with the submitted rows 99.85% on 891 rows (bar 98.5%). A separate model from Bespoke Nimble 9B v3, which stays on the board. License CC BY-NC 4.0. Decision Index 0.4.1 (release-v4), source generated 2026-10-29T14:04:16Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Source notes: New in the 2026-10-20 PR batch (PR #226). Submitted public 0.3 run adopted (kit rescore 68.46, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving; decision agreement with the submitted rows 99.85% on 891 rows (bar 98.5%). A separate model from Bespoke Nimble 9B v3, which stays on the board. License CC BY-NC 4.0. Author public-run adoption/execution status and private lab setup are disclosed in configuration; evaluator-published does not mean every public run was independently executed. Area-specific item counts unpublished; do not use whole-sample count. The board does not name a served checkpoint in this repository history. Decision Index 0.4 (release-v4), source generated 2026-10-20T14:13:56Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Checked / reported: 2026-11-02 · Samples: Not published Snapshot: 2026-10-20T14:13:56Z · Revision: 133313cf8963f2b6bed960abb0152fd14cbb6731; c130faf65f82e81d3bdefb01e758c712dc5c5199 for the 2 November 2026 additions · Published observations; result reuse terms not specified Result source · Benchmark source
HF text calibration 0.4 · Retrieval · ECE	2.02 pp	Setup & source HF text calibration 0.4 · Retrieval · ECE Exact source engine nimble-v4-max; {"kind": "full fine-tune", "technique": "autoregressive", "technique_name": "autoregressive", "served_params": 27356728560, "params_estimated": false, "params_from_name": false, "trained_bytes": null, "model_url": "https://huggingface.co/bespokelabs/Nimble-v4-max", "code_url": null, "base_model": "Qwen/Qwen3.8-27B", "base_url": "https://huggingface.co/Qwen/Qwen3.8-27B", "weights_repo": null, "served_checkpoint": null, "org": {"id": "bespokelabs", "name": "Bespoke Labs", "logo": "data/orgs/bespokelabs.png"}, "pr": 226, "new": true}. New in the 2026-10-20 PR batch (PR #226). Submitted public 0.3 run adopted (kit rescore 68.46, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving; decision agreement with the submitted rows 99.85% on 891 rows (bar 98.5%). A separate model from Bespoke Nimble 9B v3, which stays on the board. License CC BY-NC 4.0. Decision Index 0.4.1 (release-v4), source generated 2026-10-29T14:04:16Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Source notes: New in the 2026-10-20 PR batch (PR #226). Submitted public 0.3 run adopted (kit rescore 68.46, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving; decision agreement with the submitted rows 99.85% on 891 rows (bar 98.5%). A separate model from Bespoke Nimble 9B v3, which stays on the board. License CC BY-NC 4.0. Author public-run adoption/execution status and private lab setup are disclosed in configuration; evaluator-published does not mean every public run was independently executed. Area-specific item counts unpublished; do not use whole-sample count. The board does not name a served checkpoint in this repository history. Decision Index 0.4 (release-v4), source generated 2026-10-20T14:13:56Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Checked / reported: 2026-11-02 · Samples: Not published Snapshot: 2026-10-20T14:13:56Z · Revision: 133313cf8963f2b6bed960abb0152fd14cbb6731; c130faf65f82e81d3bdefb01e758c712dc5c5199 for the 2 November 2026 additions · Published observations; result reuse terms not specified Result source · Benchmark source
HF text calibration 0.4 · Retrieval · Sample accuracy	79.82%	Setup & source HF text calibration 0.4 · Retrieval · Sample accuracy Exact source engine nimble-v4-max; {"kind": "full fine-tune", "technique": "autoregressive", "technique_name": "autoregressive", "served_params": 27356728560, "params_estimated": false, "params_from_name": false, "trained_bytes": null, "model_url": "https://huggingface.co/bespokelabs/Nimble-v4-max", "code_url": null, "base_model": "Qwen/Qwen3.8-27B", "base_url": "https://huggingface.co/Qwen/Qwen3.8-27B", "weights_repo": null, "served_checkpoint": null, "org": {"id": "bespokelabs", "name": "Bespoke Labs", "logo": "data/orgs/bespokelabs.png"}, "pr": 226, "new": true}. New in the 2026-10-20 PR batch (PR #226). Submitted public 0.3 run adopted (kit rescore 68.46, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving; decision agreement with the submitted rows 99.85% on 891 rows (bar 98.5%). A separate model from Bespoke Nimble 9B v3, which stays on the board. License CC BY-NC 4.0. Decision Index 0.4.1 (release-v4), source generated 2026-10-29T14:04:16Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Source notes: New in the 2026-10-20 PR batch (PR #226). Submitted public 0.3 run adopted (kit rescore 68.46, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving; decision agreement with the submitted rows 99.85% on 891 rows (bar 98.5%). A separate model from Bespoke Nimble 9B v3, which stays on the board. License CC BY-NC 4.0. Author public-run adoption/execution status and private lab setup are disclosed in configuration; evaluator-published does not mean every public run was independently executed. Area-specific item counts unpublished; do not use whole-sample count. The board does not name a served checkpoint in this repository history. Decision Index 0.4 (release-v4), source generated 2026-10-20T14:13:56Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Checked / reported: 2026-11-02 · Samples: Not published Snapshot: 2026-10-20T14:13:56Z · Revision: 133313cf8963f2b6bed960abb0152fd14cbb6731; c130faf65f82e81d3bdefb01e758c712dc5c5199 for the 2 November 2026 additions · Published observations; result reuse terms not specified Result source · Benchmark source
HF text calibration 0.4 · Retrieval · Mean chosen confidence	79.83%	Setup & source HF text calibration 0.4 · Retrieval · Mean chosen confidence Exact source engine nimble-v4-max; {"kind": "full fine-tune", "technique": "autoregressive", "technique_name": "autoregressive", "served_params": 27356728560, "params_estimated": false, "params_from_name": false, "trained_bytes": null, "model_url": "https://huggingface.co/bespokelabs/Nimble-v4-max", "code_url": null, "base_model": "Qwen/Qwen3.8-27B", "base_url": "https://huggingface.co/Qwen/Qwen3.8-27B", "weights_repo": null, "served_checkpoint": null, "org": {"id": "bespokelabs", "name": "Bespoke Labs", "logo": "data/orgs/bespokelabs.png"}, "pr": 226, "new": true}. New in the 2026-10-20 PR batch (PR #226). Submitted public 0.3 run adopted (kit rescore 68.46, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving; decision agreement with the submitted rows 99.85% on 891 rows (bar 98.5%). A separate model from Bespoke Nimble 9B v3, which stays on the board. License CC BY-NC 4.0. Decision Index 0.4.1 (release-v4), source generated 2026-10-29T14:04:16Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Source notes: New in the 2026-10-20 PR batch (PR #226). Submitted public 0.3 run adopted (kit rescore 68.46, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving; decision agreement with the submitted rows 99.85% on 891 rows (bar 98.5%). A separate model from Bespoke Nimble 9B v3, which stays on the board. License CC BY-NC 4.0. Author public-run adoption/execution status and private lab setup are disclosed in configuration; evaluator-published does not mean every public run was independently executed. Area-specific item counts unpublished; do not use whole-sample count. The board does not name a served checkpoint in this repository history. Decision Index 0.4 (release-v4), source generated 2026-10-20T14:13:56Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Checked / reported: 2026-11-02 · Samples: Not published Snapshot: 2026-10-20T14:13:56Z · Revision: 133313cf8963f2b6bed960abb0152fd14cbb6731; c130faf65f82e81d3bdefb01e758c712dc5c5199 for the 2 November 2026 additions · Published observations; result reuse terms not specified Result source · Benchmark source