mammoth-1 Decision Model Results

mammoth-1

Explore mammoth-1: exact decision configuration, published specifications, source-linked benchmarks and API tariffs when available.

Published specifications

Provider: Baseten
Version: baselabs/mammoth-1 (27B)
System type: Trained checkpoint
Access: Open weights
License: MIT
Inputs: text, image
Decision types: Choice, Yes / no, Score
Published size: 27.36B
Active parameters: Not published
Context / input limit: 16,384 tokens (max sequence)
Announced: 2026-10-21

Hugging Face · Announcement · Website · OpenRouter

Model notes

Base Labs is the research team at Baseten. Fine-tune of Qwen3.5-27B on the 6.21M-row Mix 8 with the vision encoder frozen; MIT license. Max sequence 16,384 tokens. Baseten's API name `decision-models/baseten-mammoth-27b-latest` is a moving alias, so results through it are not pinned to a commit. OpenRouter endpoint label baseten/mammoth-1-20261021. Weights committed 19 to 20 October 2026. Typed outputs are probabilities over supplied options; confidence is not a correctness guarantee.

Checkpoint revision: 61cec44c281c693992e3d6ebeb44dd3402dfa186

← All decision models · Compare this decision system →

Published API tariff

$0.1 per 1M input tokens

Baseten · checked 2026-11-02

Baseten library page: output and cached input show $0.00. Free for a launch week, which ended 27 October 2026.

No output-token charge does not mean free requests. These are provider tariffs, separate from benchmark workload costs.

388 recorded observations. Published results retain their source and setup; no Examenos evaluation was run. Missing results are not zero. How to read these comparisons →

Official / vendor · Base Labs model cards (vendor reported)

mammoth-1 · Official / vendor · Base Labs model cards (vendor reported)
Evaluation	Result	Evidence
Base Labs cards · Jev Decision Index 0.4 Full	71.24	Setup & source Full score as printed in the cards, #1 of 208 for mammoth-1 when written. Printed in the mammoth-1 and elephant-1 cards. Printed in the vendor's own table. Base Labs' own figures from the model cards and report. Its Decision Index values are from the 0.4 board when the report was written, so they differ from the community 0.4.1 rows here. Its Artificial Analysis replica is a vendor rerun and is never an AA row. Not ranked here. Checked / reported: 2026-11-02 · Samples: Not published Snapshot: 2026-10-21 · Revision: mammoth 61cec44c28 elephant b3f8a93462 joey 9efbe485ef · Published observations; result reuse terms not specified Result source · Benchmark source
Base Labs cards · Replica of the Artificial Analysis Decision Index	65.9	Setup & source Base Labs' own replica of AA's index. AA's published values are stored separately. Base Labs' own replica; not an Artificial Analysis evaluation. Printed in the vendor's own table. Base Labs' own figures from the model cards and report. Its Decision Index values are from the 0.4 board when the report was written, so they differ from the community 0.4.1 rows here. Its Artificial Analysis replica is a vendor rerun and is never an AA row. Not ranked here. Checked / reported: 2026-11-02 · Samples: Not published Snapshot: 2026-10-21 · Revision: mammoth 61cec44c28 elephant b3f8a93462 joey 9efbe485ef · Published observations; result reuse terms not specified Result source · Benchmark source

Evaluator-published · Community Decision Index

Evaluator-published · etsabary System 1 Decision Benchmark

mammoth-1 · Evaluator-published · etsabary System 1 Decision Benchmark
Evaluation	Result	Evidence
System 1 Decision Benchmark v2 · All-question accuracy	80.662%	Setup & source Correct answers over all 1,934 questions. Mammoth-1 (Baseten direct; latest named alias); Hosted native; returned identities {"decision-models/baseten-mammoth-27b-latest": 1933, "unreported in selected primary": 1}; collected 2026-10-21 to 2026-10-31 America/Toronto; one primary response per input, no scored retries. Unsuccessful responses counted as wrong: {"service/rate limit": 1}. Development benchmark, not an independently sampled test: 1,934 questions in 41 categories, the frozen 1,342-question bank plus 192 logical and 400 practical additions. The bank was built with model-guided development, and Jev plus retained Microsoft and Drex feedback are development exposure. Related variants and a single recorded primary response per input limit generalization. The scoring key stays private, so public checks cover coverage, timing and aggregate consistency but cannot rescore answers. Strict accuracy counts a wrong usable answer and an unsuccessful response in the same fixed denominator. Timing pools primary client times with the network included; providers, precision, concurrency, routes and dates differ, retained and new timings can span sessions, and quick failures can reduce all-primary latency. The source published a separate cost ledger on 1 November 2026, stored as decision-reference-v2-cost-per-1000, which supersedes its earlier “cost not consolidated” note. Older 1,342-only scores are not converted into this bank. Not ranked here. Checked / reported: 2026-11-02 · Samples: 1934 Snapshot: 2026-11-01 · Revision: f5fa9cfafccd97fcfbdb0336e14284faaa8d2d35 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
System 1 Decision Benchmark v2 · Macro-category accuracy	80.399%	Setup & source Equal-weight mean over the 41 categories. Mammoth-1 (Baseten direct; latest named alias); Hosted native; returned identities {"decision-models/baseten-mammoth-27b-latest": 1933, "unreported in selected primary": 1}; collected 2026-10-21 to 2026-10-31 America/Toronto; one primary response per input, no scored retries. Unsuccessful responses counted as wrong: {"service/rate limit": 1}. Development benchmark, not an independently sampled test: 1,934 questions in 41 categories, the frozen 1,342-question bank plus 192 logical and 400 practical additions. The bank was built with model-guided development, and Jev plus retained Microsoft and Drex feedback are development exposure. Related variants and a single recorded primary response per input limit generalization. The scoring key stays private, so public checks cover coverage, timing and aggregate consistency but cannot rescore answers. Strict accuracy counts a wrong usable answer and an unsuccessful response in the same fixed denominator. Timing pools primary client times with the network included; providers, precision, concurrency, routes and dates differ, retained and new timings can span sessions, and quick failures can reduce all-primary latency. The source published a separate cost ledger on 1 November 2026, stored as decision-reference-v2-cost-per-1000, which supersedes its earlier “cost not consolidated” note. Older 1,342-only scores are not converted into this bank. Not ranked here. Checked / reported: 2026-11-02 · Samples: 1934 Snapshot: 2026-11-01 · Revision: f5fa9cfafccd97fcfbdb0336e14284faaa8d2d35 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
System 1 Decision Benchmark v2 · Added 592 questions	71.115%	Setup & source Accuracy on the 592 questions new in v2. Mammoth-1 (Baseten direct; latest named alias); Hosted native; returned identities {"decision-models/baseten-mammoth-27b-latest": 1933, "unreported in selected primary": 1}; collected 2026-10-21 to 2026-10-31 America/Toronto; one primary response per input, no scored retries. Unsuccessful responses counted as wrong: {"service/rate limit": 1}. Development benchmark, not an independently sampled test: 1,934 questions in 41 categories, the frozen 1,342-question bank plus 192 logical and 400 practical additions. The bank was built with model-guided development, and Jev plus retained Microsoft and Drex feedback are development exposure. Related variants and a single recorded primary response per input limit generalization. The scoring key stays private, so public checks cover coverage, timing and aggregate consistency but cannot rescore answers. Strict accuracy counts a wrong usable answer and an unsuccessful response in the same fixed denominator. Timing pools primary client times with the network included; providers, precision, concurrency, routes and dates differ, retained and new timings can span sessions, and quick failures can reduce all-primary latency. The source published a separate cost ledger on 1 November 2026, stored as decision-reference-v2-cost-per-1000, which supersedes its earlier “cost not consolidated” note. Older 1,342-only scores are not converted into this bank. Not ranked here. Checked / reported: 2026-11-02 · Samples: 592 Snapshot: 2026-11-01 · Revision: f5fa9cfafccd97fcfbdb0336e14284faaa8d2d35 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
System 1 Decision Benchmark v2 · Logical additions	62.500%	Setup & source Accuracy on the 192 logical additions. Mammoth-1 (Baseten direct; latest named alias); Hosted native; returned identities {"decision-models/baseten-mammoth-27b-latest": 1933, "unreported in selected primary": 1}; collected 2026-10-21 to 2026-10-31 America/Toronto; one primary response per input, no scored retries. Unsuccessful responses counted as wrong: {"service/rate limit": 1}. Development benchmark, not an independently sampled test: 1,934 questions in 41 categories, the frozen 1,342-question bank plus 192 logical and 400 practical additions. The bank was built with model-guided development, and Jev plus retained Microsoft and Drex feedback are development exposure. Related variants and a single recorded primary response per input limit generalization. The scoring key stays private, so public checks cover coverage, timing and aggregate consistency but cannot rescore answers. Strict accuracy counts a wrong usable answer and an unsuccessful response in the same fixed denominator. Timing pools primary client times with the network included; providers, precision, concurrency, routes and dates differ, retained and new timings can span sessions, and quick failures can reduce all-primary latency. The source published a separate cost ledger on 1 November 2026, stored as decision-reference-v2-cost-per-1000, which supersedes its earlier “cost not consolidated” note. Older 1,342-only scores are not converted into this bank. Not ranked here. Checked / reported: 2026-11-02 · Samples: 192 Snapshot: 2026-11-01 · Revision: f5fa9cfafccd97fcfbdb0336e14284faaa8d2d35 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
System 1 Decision Benchmark v2 · Practical additions	75.250%	Setup & source Accuracy on the 400 practical additions. Mammoth-1 (Baseten direct; latest named alias); Hosted native; returned identities {"decision-models/baseten-mammoth-27b-latest": 1933, "unreported in selected primary": 1}; collected 2026-10-21 to 2026-10-31 America/Toronto; one primary response per input, no scored retries. Unsuccessful responses counted as wrong: {"service/rate limit": 1}. Development benchmark, not an independently sampled test: 1,934 questions in 41 categories, the frozen 1,342-question bank plus 192 logical and 400 practical additions. The bank was built with model-guided development, and Jev plus retained Microsoft and Drex feedback are development exposure. Related variants and a single recorded primary response per input limit generalization. The scoring key stays private, so public checks cover coverage, timing and aggregate consistency but cannot rescore answers. Strict accuracy counts a wrong usable answer and an unsuccessful response in the same fixed denominator. Timing pools primary client times with the network included; providers, precision, concurrency, routes and dates differ, retained and new timings can span sessions, and quick failures can reduce all-primary latency. The source published a separate cost ledger on 1 November 2026, stored as decision-reference-v2-cost-per-1000, which supersedes its earlier “cost not consolidated” note. Older 1,342-only scores are not converted into this bank. Not ranked here. Checked / reported: 2026-11-02 · Samples: 400 Snapshot: 2026-11-01 · Revision: f5fa9cfafccd97fcfbdb0336e14284faaa8d2d35 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
System 1 Decision Benchmark v2 · Original 1,342 questions	84.873%	Setup & source Accuracy on the frozen original inputs inside the v2 bank. Mammoth-1 (Baseten direct; latest named alias); Hosted native; returned identities {"decision-models/baseten-mammoth-27b-latest": 1933, "unreported in selected primary": 1}; collected 2026-10-21 to 2026-10-31 America/Toronto; one primary response per input, no scored retries. Unsuccessful responses counted as wrong: {"service/rate limit": 1}. Development benchmark, not an independently sampled test: 1,934 questions in 41 categories, the frozen 1,342-question bank plus 192 logical and 400 practical additions. The bank was built with model-guided development, and Jev plus retained Microsoft and Drex feedback are development exposure. Related variants and a single recorded primary response per input limit generalization. The scoring key stays private, so public checks cover coverage, timing and aggregate consistency but cannot rescore answers. Strict accuracy counts a wrong usable answer and an unsuccessful response in the same fixed denominator. Timing pools primary client times with the network included; providers, precision, concurrency, routes and dates differ, retained and new timings can span sessions, and quick failures can reduce all-primary latency. The source published a separate cost ledger on 1 November 2026, stored as decision-reference-v2-cost-per-1000, which supersedes its earlier “cost not consolidated” note. Older 1,342-only scores are not converted into this bank. Not ranked here. Checked / reported: 2026-11-02 · Samples: 1342 Snapshot: 2026-11-01 · Revision: f5fa9cfafccd97fcfbdb0336e14284faaa8d2d35 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
System 1 Decision Benchmark v2 · Median client time	202.8 ms	Setup & source Pooled primary complete-response client time, network included. Mammoth-1 (Baseten direct; latest named alias); Hosted native; returned identities {"decision-models/baseten-mammoth-27b-latest": 1933, "unreported in selected primary": 1}; collected 2026-10-21 to 2026-10-31 America/Toronto; one primary response per input, no scored retries. Unsuccessful responses counted as wrong: {"service/rate limit": 1}. Development benchmark, not an independently sampled test: 1,934 questions in 41 categories, the frozen 1,342-question bank plus 192 logical and 400 practical additions. The bank was built with model-guided development, and Jev plus retained Microsoft and Drex feedback are development exposure. Related variants and a single recorded primary response per input limit generalization. The scoring key stays private, so public checks cover coverage, timing and aggregate consistency but cannot rescore answers. Strict accuracy counts a wrong usable answer and an unsuccessful response in the same fixed denominator. Timing pools primary client times with the network included; providers, precision, concurrency, routes and dates differ, retained and new timings can span sessions, and quick failures can reduce all-primary latency. The source published a separate cost ledger on 1 November 2026, stored as decision-reference-v2-cost-per-1000, which supersedes its earlier “cost not consolidated” note. Older 1,342-only scores are not converted into this bank. Not ranked here. Checked / reported: 2026-11-02 · Samples: 1934 Snapshot: 2026-11-01 · Revision: f5fa9cfafccd97fcfbdb0336e14284faaa8d2d35 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
System 1 Decision Benchmark v2 · P95 client time	629.6 ms	Setup & source Pooled primary complete-response client time, network included. Mammoth-1 (Baseten direct; latest named alias); Hosted native; returned identities {"decision-models/baseten-mammoth-27b-latest": 1933, "unreported in selected primary": 1}; collected 2026-10-21 to 2026-10-31 America/Toronto; one primary response per input, no scored retries. Unsuccessful responses counted as wrong: {"service/rate limit": 1}. Development benchmark, not an independently sampled test: 1,934 questions in 41 categories, the frozen 1,342-question bank plus 192 logical and 400 practical additions. The bank was built with model-guided development, and Jev plus retained Microsoft and Drex feedback are development exposure. Related variants and a single recorded primary response per input limit generalization. The scoring key stays private, so public checks cover coverage, timing and aggregate consistency but cannot rescore answers. Strict accuracy counts a wrong usable answer and an unsuccessful response in the same fixed denominator. Timing pools primary client times with the network included; providers, precision, concurrency, routes and dates differ, retained and new timings can span sessions, and quick failures can reduce all-primary latency. The source published a separate cost ledger on 1 November 2026, stored as decision-reference-v2-cost-per-1000, which supersedes its earlier “cost not consolidated” note. Older 1,342-only scores are not converted into this bank. Not ranked here. Checked / reported: 2026-11-02 · Samples: 1934 Snapshot: 2026-11-01 · Revision: f5fa9cfafccd97fcfbdb0336e14284faaa8d2d35 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
System 1 Decision Benchmark v2 · F01 · Lookup and filtering	93.023%	Setup & source Lookup and filtering: 43 questions. Mammoth-1 (Baseten direct; latest named alias); Hosted native; returned identities {"decision-models/baseten-mammoth-27b-latest": 1933, "unreported in selected primary": 1}; collected 2026-10-21 to 2026-10-31 America/Toronto; one primary response per input, no scored retries. Unsuccessful responses counted as wrong: {"service/rate limit": 1}. Development benchmark, not an independently sampled test: 1,934 questions in 41 categories, the frozen 1,342-question bank plus 192 logical and 400 practical additions. The bank was built with model-guided development, and Jev plus retained Microsoft and Drex feedback are development exposure. Related variants and a single recorded primary response per input limit generalization. The scoring key stays private, so public checks cover coverage, timing and aggregate consistency but cannot rescore answers. Strict accuracy counts a wrong usable answer and an unsuccessful response in the same fixed denominator. Timing pools primary client times with the network included; providers, precision, concurrency, routes and dates differ, retained and new timings can span sessions, and quick failures can reduce all-primary latency. The source published a separate cost ledger on 1 November 2026, stored as decision-reference-v2-cost-per-1000, which supersedes its earlier “cost not consolidated” note. Older 1,342-only scores are not converted into this bank. Not ranked here. Checked / reported: 2026-11-02 · Samples: 43 Snapshot: 2026-11-01 · Revision: f5fa9cfafccd97fcfbdb0336e14284faaa8d2d35 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
System 1 Decision Benchmark v2 · F02 · Forward conditionals	97.826%	Setup & source Forward conditionals: 46 questions. Mammoth-1 (Baseten direct; latest named alias); Hosted native; returned identities {"decision-models/baseten-mammoth-27b-latest": 1933, "unreported in selected primary": 1}; collected 2026-10-21 to 2026-10-31 America/Toronto; one primary response per input, no scored retries. Unsuccessful responses counted as wrong: {"service/rate limit": 1}. Development benchmark, not an independently sampled test: 1,934 questions in 41 categories, the frozen 1,342-question bank plus 192 logical and 400 practical additions. The bank was built with model-guided development, and Jev plus retained Microsoft and Drex feedback are development exposure. Related variants and a single recorded primary response per input limit generalization. The scoring key stays private, so public checks cover coverage, timing and aggregate consistency but cannot rescore answers. Strict accuracy counts a wrong usable answer and an unsuccessful response in the same fixed denominator. Timing pools primary client times with the network included; providers, precision, concurrency, routes and dates differ, retained and new timings can span sessions, and quick failures can reduce all-primary latency. The source published a separate cost ledger on 1 November 2026, stored as decision-reference-v2-cost-per-1000, which supersedes its earlier “cost not consolidated” note. Older 1,342-only scores are not converted into this bank. Not ranked here. Checked / reported: 2026-11-02 · Samples: 46 Snapshot: 2026-11-01 · Revision: f5fa9cfafccd97fcfbdb0336e14284faaa8d2d35 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
System 1 Decision Benchmark v2 · F03 · Necessary conditions	96.154%	Setup & source Necessary conditions: 52 questions. Mammoth-1 (Baseten direct; latest named alias); Hosted native; returned identities {"decision-models/baseten-mammoth-27b-latest": 1933, "unreported in selected primary": 1}; collected 2026-10-21 to 2026-10-31 America/Toronto; one primary response per input, no scored retries. Unsuccessful responses counted as wrong: {"service/rate limit": 1}. Development benchmark, not an independently sampled test: 1,934 questions in 41 categories, the frozen 1,342-question bank plus 192 logical and 400 practical additions. The bank was built with model-guided development, and Jev plus retained Microsoft and Drex feedback are development exposure. Related variants and a single recorded primary response per input limit generalization. The scoring key stays private, so public checks cover coverage, timing and aggregate consistency but cannot rescore answers. Strict accuracy counts a wrong usable answer and an unsuccessful response in the same fixed denominator. Timing pools primary client times with the network included; providers, precision, concurrency, routes and dates differ, retained and new timings can span sessions, and quick failures can reduce all-primary latency. The source published a separate cost ledger on 1 November 2026, stored as decision-reference-v2-cost-per-1000, which supersedes its earlier “cost not consolidated” note. Older 1,342-only scores are not converted into this bank. Not ranked here. Checked / reported: 2026-11-02 · Samples: 52 Snapshot: 2026-11-01 · Revision: f5fa9cfafccd97fcfbdb0336e14284faaa8d2d35 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
System 1 Decision Benchmark v2 · F04 · Boolean scope	94.444%	Setup & source Boolean scope: 54 questions. Mammoth-1 (Baseten direct; latest named alias); Hosted native; returned identities {"decision-models/baseten-mammoth-27b-latest": 1933, "unreported in selected primary": 1}; collected 2026-10-21 to 2026-10-31 America/Toronto; one primary response per input, no scored retries. Unsuccessful responses counted as wrong: {"service/rate limit": 1}. Development benchmark, not an independently sampled test: 1,934 questions in 41 categories, the frozen 1,342-question bank plus 192 logical and 400 practical additions. The bank was built with model-guided development, and Jev plus retained Microsoft and Drex feedback are development exposure. Related variants and a single recorded primary response per input limit generalization. The scoring key stays private, so public checks cover coverage, timing and aggregate consistency but cannot rescore answers. Strict accuracy counts a wrong usable answer and an unsuccessful response in the same fixed denominator. Timing pools primary client times with the network included; providers, precision, concurrency, routes and dates differ, retained and new timings can span sessions, and quick failures can reduce all-primary latency. The source published a separate cost ledger on 1 November 2026, stored as decision-reference-v2-cost-per-1000, which supersedes its earlier “cost not consolidated” note. Older 1,342-only scores are not converted into this bank. Not ranked here. Checked / reported: 2026-11-02 · Samples: 54 Snapshot: 2026-11-01 · Revision: f5fa9cfafccd97fcfbdb0336e14284faaa8d2d35 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
System 1 Decision Benchmark v2 · F05 · Quantifiers and sets	87.755%	Setup & source Quantifiers and sets: 49 questions. Mammoth-1 (Baseten direct; latest named alias); Hosted native; returned identities {"decision-models/baseten-mammoth-27b-latest": 1933, "unreported in selected primary": 1}; collected 2026-10-21 to 2026-10-31 America/Toronto; one primary response per input, no scored retries. Unsuccessful responses counted as wrong: {"service/rate limit": 1}. Development benchmark, not an independently sampled test: 1,934 questions in 41 categories, the frozen 1,342-question bank plus 192 logical and 400 practical additions. The bank was built with model-guided development, and Jev plus retained Microsoft and Drex feedback are development exposure. Related variants and a single recorded primary response per input limit generalization. The scoring key stays private, so public checks cover coverage, timing and aggregate consistency but cannot rescore answers. Strict accuracy counts a wrong usable answer and an unsuccessful response in the same fixed denominator. Timing pools primary client times with the network included; providers, precision, concurrency, routes and dates differ, retained and new timings can span sessions, and quick failures can reduce all-primary latency. The source published a separate cost ledger on 1 November 2026, stored as decision-reference-v2-cost-per-1000, which supersedes its earlier “cost not consolidated” note. Older 1,342-only scores are not converted into this bank. Not ranked here. Checked / reported: 2026-11-02 · Samples: 49 Snapshot: 2026-11-01 · Revision: f5fa9cfafccd97fcfbdb0336e14284faaa8d2d35 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
System 1 Decision Benchmark v2 · F06 · Comparisons	97.500%	Setup & source Comparisons: 40 questions. Mammoth-1 (Baseten direct; latest named alias); Hosted native; returned identities {"decision-models/baseten-mammoth-27b-latest": 1933, "unreported in selected primary": 1}; collected 2026-10-21 to 2026-10-31 America/Toronto; one primary response per input, no scored retries. Unsuccessful responses counted as wrong: {"service/rate limit": 1}. Development benchmark, not an independently sampled test: 1,934 questions in 41 categories, the frozen 1,342-question bank plus 192 logical and 400 practical additions. The bank was built with model-guided development, and Jev plus retained Microsoft and Drex feedback are development exposure. Related variants and a single recorded primary response per input limit generalization. The scoring key stays private, so public checks cover coverage, timing and aggregate consistency but cannot rescore answers. Strict accuracy counts a wrong usable answer and an unsuccessful response in the same fixed denominator. Timing pools primary client times with the network included; providers, precision, concurrency, routes and dates differ, retained and new timings can span sessions, and quick failures can reduce all-primary latency. The source published a separate cost ledger on 1 November 2026, stored as decision-reference-v2-cost-per-1000, which supersedes its earlier “cost not consolidated” note. Older 1,342-only scores are not converted into this bank. Not ranked here. Checked / reported: 2026-11-02 · Samples: 40 Snapshot: 2026-11-01 · Revision: f5fa9cfafccd97fcfbdb0336e14284faaa8d2d35 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
System 1 Decision Benchmark v2 · F07 · Scheduling	77.632%	Setup & source Scheduling: 76 questions. Mammoth-1 (Baseten direct; latest named alias); Hosted native; returned identities {"decision-models/baseten-mammoth-27b-latest": 1933, "unreported in selected primary": 1}; collected 2026-10-21 to 2026-10-31 America/Toronto; one primary response per input, no scored retries. Unsuccessful responses counted as wrong: {"service/rate limit": 1}. Development benchmark, not an independently sampled test: 1,934 questions in 41 categories, the frozen 1,342-question bank plus 192 logical and 400 practical additions. The bank was built with model-guided development, and Jev plus retained Microsoft and Drex feedback are development exposure. Related variants and a single recorded primary response per input limit generalization. The scoring key stays private, so public checks cover coverage, timing and aggregate consistency but cannot rescore answers. Strict accuracy counts a wrong usable answer and an unsuccessful response in the same fixed denominator. Timing pools primary client times with the network included; providers, precision, concurrency, routes and dates differ, retained and new timings can span sessions, and quick failures can reduce all-primary latency. The source published a separate cost ledger on 1 November 2026, stored as decision-reference-v2-cost-per-1000, which supersedes its earlier “cost not consolidated” note. Older 1,342-only scores are not converted into this bank. Not ranked here. Checked / reported: 2026-11-02 · Samples: 76 Snapshot: 2026-11-01 · Revision: f5fa9cfafccd97fcfbdb0336e14284faaa8d2d35 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
System 1 Decision Benchmark v2 · F08 · One-to-one assignments	95.522%	Setup & source One-to-one assignments: 67 questions. Mammoth-1 (Baseten direct; latest named alias); Hosted native; returned identities {"decision-models/baseten-mammoth-27b-latest": 1933, "unreported in selected primary": 1}; collected 2026-10-21 to 2026-10-31 America/Toronto; one primary response per input, no scored retries. Unsuccessful responses counted as wrong: {"service/rate limit": 1}. Development benchmark, not an independently sampled test: 1,934 questions in 41 categories, the frozen 1,342-question bank plus 192 logical and 400 practical additions. The bank was built with model-guided development, and Jev plus retained Microsoft and Drex feedback are development exposure. Related variants and a single recorded primary response per input limit generalization. The scoring key stays private, so public checks cover coverage, timing and aggregate consistency but cannot rescore answers. Strict accuracy counts a wrong usable answer and an unsuccessful response in the same fixed denominator. Timing pools primary client times with the network included; providers, precision, concurrency, routes and dates differ, retained and new timings can span sessions, and quick failures can reduce all-primary latency. The source published a separate cost ledger on 1 November 2026, stored as decision-reference-v2-cost-per-1000, which supersedes its earlier “cost not consolidated” note. Older 1,342-only scores are not converted into this bank. Not ranked here. Checked / reported: 2026-11-02 · Samples: 67 Snapshot: 2026-11-01 · Revision: f5fa9cfafccd97fcfbdb0336e14284faaa8d2d35 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
System 1 Decision Benchmark v2 · F09 · Exact truth counting	61.972%	Setup & source Exact truth counting: 71 questions. Mammoth-1 (Baseten direct; latest named alias); Hosted native; returned identities {"decision-models/baseten-mammoth-27b-latest": 1933, "unreported in selected primary": 1}; collected 2026-10-21 to 2026-10-31 America/Toronto; one primary response per input, no scored retries. Unsuccessful responses counted as wrong: {"service/rate limit": 1}. Development benchmark, not an independently sampled test: 1,934 questions in 41 categories, the frozen 1,342-question bank plus 192 logical and 400 practical additions. The bank was built with model-guided development, and Jev plus retained Microsoft and Drex feedback are development exposure. Related variants and a single recorded primary response per input limit generalization. The scoring key stays private, so public checks cover coverage, timing and aggregate consistency but cannot rescore answers. Strict accuracy counts a wrong usable answer and an unsuccessful response in the same fixed denominator. Timing pools primary client times with the network included; providers, precision, concurrency, routes and dates differ, retained and new timings can span sessions, and quick failures can reduce all-primary latency. The source published a separate cost ledger on 1 November 2026, stored as decision-reference-v2-cost-per-1000, which supersedes its earlier “cost not consolidated” note. Older 1,342-only scores are not converted into this bank. Not ranked here. Checked / reported: 2026-11-02 · Samples: 71 Snapshot: 2026-11-01 · Revision: f5fa9cfafccd97fcfbdb0336e14284faaa8d2d35 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
System 1 Decision Benchmark v2 · F10 · Truth-teller consistency	64.865%	Setup & source Truth-teller consistency: 74 questions. Mammoth-1 (Baseten direct; latest named alias); Hosted native; returned identities {"decision-models/baseten-mammoth-27b-latest": 1933, "unreported in selected primary": 1}; collected 2026-10-21 to 2026-10-31 America/Toronto; one primary response per input, no scored retries. Unsuccessful responses counted as wrong: {"service/rate limit": 1}. Development benchmark, not an independently sampled test: 1,934 questions in 41 categories, the frozen 1,342-question bank plus 192 logical and 400 practical additions. The bank was built with model-guided development, and Jev plus retained Microsoft and Drex feedback are development exposure. Related variants and a single recorded primary response per input limit generalization. The scoring key stays private, so public checks cover coverage, timing and aggregate consistency but cannot rescore answers. Strict accuracy counts a wrong usable answer and an unsuccessful response in the same fixed denominator. Timing pools primary client times with the network included; providers, precision, concurrency, routes and dates differ, retained and new timings can span sessions, and quick failures can reduce all-primary latency. The source published a separate cost ledger on 1 November 2026, stored as decision-reference-v2-cost-per-1000, which supersedes its earlier “cost not consolidated” note. Older 1,342-only scores are not converted into this bank. Not ranked here. Checked / reported: 2026-11-02 · Samples: 74 Snapshot: 2026-11-01 · Revision: f5fa9cfafccd97fcfbdb0336e14284faaa8d2d35 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
System 1 Decision Benchmark v2 · F11 · Sufficiency and inconsistency	92.683%	Setup & source Sufficiency and inconsistency: 41 questions. Mammoth-1 (Baseten direct; latest named alias); Hosted native; returned identities {"decision-models/baseten-mammoth-27b-latest": 1933, "unreported in selected primary": 1}; collected 2026-10-21 to 2026-10-31 America/Toronto; one primary response per input, no scored retries. Unsuccessful responses counted as wrong: {"service/rate limit": 1}. Development benchmark, not an independently sampled test: 1,934 questions in 41 categories, the frozen 1,342-question bank plus 192 logical and 400 practical additions. The bank was built with model-guided development, and Jev plus retained Microsoft and Drex feedback are development exposure. Related variants and a single recorded primary response per input limit generalization. The scoring key stays private, so public checks cover coverage, timing and aggregate consistency but cannot rescore answers. Strict accuracy counts a wrong usable answer and an unsuccessful response in the same fixed denominator. Timing pools primary client times with the network included; providers, precision, concurrency, routes and dates differ, retained and new timings can span sessions, and quick failures can reduce all-primary latency. The source published a separate cost ledger on 1 November 2026, stored as decision-reference-v2-cost-per-1000, which supersedes its earlier “cost not consolidated” note. Older 1,342-only scores are not converted into this bank. Not ranked here. Checked / reported: 2026-11-02 · Samples: 41 Snapshot: 2026-11-01 · Revision: f5fa9cfafccd97fcfbdb0336e14284faaa8d2d35 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
System 1 Decision Benchmark v2 · F12 · Object swaps	73.077%	Setup & source Object swaps: 52 questions. Mammoth-1 (Baseten direct; latest named alias); Hosted native; returned identities {"decision-models/baseten-mammoth-27b-latest": 1933, "unreported in selected primary": 1}; collected 2026-10-21 to 2026-10-31 America/Toronto; one primary response per input, no scored retries. Unsuccessful responses counted as wrong: {"service/rate limit": 1}. Development benchmark, not an independently sampled test: 1,934 questions in 41 categories, the frozen 1,342-question bank plus 192 logical and 400 practical additions. The bank was built with model-guided development, and Jev plus retained Microsoft and Drex feedback are development exposure. Related variants and a single recorded primary response per input limit generalization. The scoring key stays private, so public checks cover coverage, timing and aggregate consistency but cannot rescore answers. Strict accuracy counts a wrong usable answer and an unsuccessful response in the same fixed denominator. Timing pools primary client times with the network included; providers, precision, concurrency, routes and dates differ, retained and new timings can span sessions, and quick failures can reduce all-primary latency. The source published a separate cost ledger on 1 November 2026, stored as decision-reference-v2-cost-per-1000, which supersedes its earlier “cost not consolidated” note. Older 1,342-only scores are not converted into this bank. Not ranked here. Checked / reported: 2026-11-02 · Samples: 52 Snapshot: 2026-11-01 · Revision: f5fa9cfafccd97fcfbdb0336e14284faaa8d2d35 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
System 1 Decision Benchmark v2 · F13 · Sequential procedures	37.500%	Setup & source Sequential procedures: 48 questions. Mammoth-1 (Baseten direct; latest named alias); Hosted native; returned identities {"decision-models/baseten-mammoth-27b-latest": 1933, "unreported in selected primary": 1}; collected 2026-10-21 to 2026-10-31 America/Toronto; one primary response per input, no scored retries. Unsuccessful responses counted as wrong: {"service/rate limit": 1}. Development benchmark, not an independently sampled test: 1,934 questions in 41 categories, the frozen 1,342-question bank plus 192 logical and 400 practical additions. The bank was built with model-guided development, and Jev plus retained Microsoft and Drex feedback are development exposure. Related variants and a single recorded primary response per input limit generalization. The scoring key stays private, so public checks cover coverage, timing and aggregate consistency but cannot rescore answers. Strict accuracy counts a wrong usable answer and an unsuccessful response in the same fixed denominator. Timing pools primary client times with the network included; providers, precision, concurrency, routes and dates differ, retained and new timings can span sessions, and quick failures can reduce all-primary latency. The source published a separate cost ledger on 1 November 2026, stored as decision-reference-v2-cost-per-1000, which supersedes its earlier “cost not consolidated” note. Older 1,342-only scores are not converted into this bank. Not ranked here. Checked / reported: 2026-11-02 · Samples: 48 Snapshot: 2026-11-01 · Revision: f5fa9cfafccd97fcfbdb0336e14284faaa8d2d35 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
System 1 Decision Benchmark v2 · F14 · Temporal intervals	95.238%	Setup & source Temporal intervals: 42 questions. Mammoth-1 (Baseten direct; latest named alias); Hosted native; returned identities {"decision-models/baseten-mammoth-27b-latest": 1933, "unreported in selected primary": 1}; collected 2026-10-21 to 2026-10-31 America/Toronto; one primary response per input, no scored retries. Unsuccessful responses counted as wrong: {"service/rate limit": 1}. Development benchmark, not an independently sampled test: 1,934 questions in 41 categories, the frozen 1,342-question bank plus 192 logical and 400 practical additions. The bank was built with model-guided development, and Jev plus retained Microsoft and Drex feedback are development exposure. Related variants and a single recorded primary response per input limit generalization. The scoring key stays private, so public checks cover coverage, timing and aggregate consistency but cannot rescore answers. Strict accuracy counts a wrong usable answer and an unsuccessful response in the same fixed denominator. Timing pools primary client times with the network included; providers, precision, concurrency, routes and dates differ, retained and new timings can span sessions, and quick failures can reduce all-primary latency. The source published a separate cost ledger on 1 November 2026, stored as decision-reference-v2-cost-per-1000, which supersedes its earlier “cost not consolidated” note. Older 1,342-only scores are not converted into this bank. Not ranked here. Checked / reported: 2026-11-02 · Samples: 42 Snapshot: 2026-11-01 · Revision: f5fa9cfafccd97fcfbdb0336e14284faaa8d2d35 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
System 1 Decision Benchmark v2 · F15 · Spatial tracking	77.551%	Setup & source Spatial tracking: 49 questions. Mammoth-1 (Baseten direct; latest named alias); Hosted native; returned identities {"decision-models/baseten-mammoth-27b-latest": 1933, "unreported in selected primary": 1}; collected 2026-10-21 to 2026-10-31 America/Toronto; one primary response per input, no scored retries. Unsuccessful responses counted as wrong: {"service/rate limit": 1}. Development benchmark, not an independently sampled test: 1,934 questions in 41 categories, the frozen 1,342-question bank plus 192 logical and 400 practical additions. The bank was built with model-guided development, and Jev plus retained Microsoft and Drex feedback are development exposure. Related variants and a single recorded primary response per input limit generalization. The scoring key stays private, so public checks cover coverage, timing and aggregate consistency but cannot rescore answers. Strict accuracy counts a wrong usable answer and an unsuccessful response in the same fixed denominator. Timing pools primary client times with the network included; providers, precision, concurrency, routes and dates differ, retained and new timings can span sessions, and quick failures can reduce all-primary latency. The source published a separate cost ledger on 1 November 2026, stored as decision-reference-v2-cost-per-1000, which supersedes its earlier “cost not consolidated” note. Older 1,342-only scores are not converted into this bank. Not ranked here. Checked / reported: 2026-11-02 · Samples: 49 Snapshot: 2026-11-01 · Revision: f5fa9cfafccd97fcfbdb0336e14284faaa8d2d35 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
System 1 Decision Benchmark v2 · F16 · Reference binding	74.000%	Setup & source Reference binding: 50 questions. Mammoth-1 (Baseten direct; latest named alias); Hosted native; returned identities {"decision-models/baseten-mammoth-27b-latest": 1933, "unreported in selected primary": 1}; collected 2026-10-21 to 2026-10-31 America/Toronto; one primary response per input, no scored retries. Unsuccessful responses counted as wrong: {"service/rate limit": 1}. Development benchmark, not an independently sampled test: 1,934 questions in 41 categories, the frozen 1,342-question bank plus 192 logical and 400 practical additions. The bank was built with model-guided development, and Jev plus retained Microsoft and Drex feedback are development exposure. Related variants and a single recorded primary response per input limit generalization. The scoring key stays private, so public checks cover coverage, timing and aggregate consistency but cannot rescore answers. Strict accuracy counts a wrong usable answer and an unsuccessful response in the same fixed denominator. Timing pools primary client times with the network included; providers, precision, concurrency, routes and dates differ, retained and new timings can span sessions, and quick failures can reduce all-primary latency. The source published a separate cost ledger on 1 November 2026, stored as decision-reference-v2-cost-per-1000, which supersedes its earlier “cost not consolidated” note. Older 1,342-only scores are not converted into this bank. Not ranked here. Checked / reported: 2026-11-02 · Samples: 50 Snapshot: 2026-11-01 · Revision: f5fa9cfafccd97fcfbdb0336e14284faaa8d2d35 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
System 1 Decision Benchmark v2 · F17 · Beliefs and perspectives	94.000%	Setup & source Beliefs and perspectives: 50 questions. Mammoth-1 (Baseten direct; latest named alias); Hosted native; returned identities {"decision-models/baseten-mammoth-27b-latest": 1933, "unreported in selected primary": 1}; collected 2026-10-21 to 2026-10-31 America/Toronto; one primary response per input, no scored retries. Unsuccessful responses counted as wrong: {"service/rate limit": 1}. Development benchmark, not an independently sampled test: 1,934 questions in 41 categories, the frozen 1,342-question bank plus 192 logical and 400 practical additions. The bank was built with model-guided development, and Jev plus retained Microsoft and Drex feedback are development exposure. Related variants and a single recorded primary response per input limit generalization. The scoring key stays private, so public checks cover coverage, timing and aggregate consistency but cannot rescore answers. Strict accuracy counts a wrong usable answer and an unsuccessful response in the same fixed denominator. Timing pools primary client times with the network included; providers, precision, concurrency, routes and dates differ, retained and new timings can span sessions, and quick failures can reduce all-primary latency. The source published a separate cost ledger on 1 November 2026, stored as decision-reference-v2-cost-per-1000, which supersedes its earlier “cost not consolidated” note. Older 1,342-only scores are not converted into this bank. Not ranked here. Checked / reported: 2026-11-02 · Samples: 50 Snapshot: 2026-11-01 · Revision: f5fa9cfafccd97fcfbdb0336e14284faaa8d2d35 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
System 1 Decision Benchmark v2 · F18 · Causal intervention	88.000%	Setup & source Causal intervention: 50 questions. Mammoth-1 (Baseten direct; latest named alias); Hosted native; returned identities {"decision-models/baseten-mammoth-27b-latest": 1933, "unreported in selected primary": 1}; collected 2026-10-21 to 2026-10-31 America/Toronto; one primary response per input, no scored retries. Unsuccessful responses counted as wrong: {"service/rate limit": 1}. Development benchmark, not an independently sampled test: 1,934 questions in 41 categories, the frozen 1,342-question bank plus 192 logical and 400 practical additions. The bank was built with model-guided development, and Jev plus retained Microsoft and Drex feedback are development exposure. Related variants and a single recorded primary response per input limit generalization. The scoring key stays private, so public checks cover coverage, timing and aggregate consistency but cannot rescore answers. Strict accuracy counts a wrong usable answer and an unsuccessful response in the same fixed denominator. Timing pools primary client times with the network included; providers, precision, concurrency, routes and dates differ, retained and new timings can span sessions, and quick failures can reduce all-primary latency. The source published a separate cost ledger on 1 November 2026, stored as decision-reference-v2-cost-per-1000, which supersedes its earlier “cost not consolidated” note. Older 1,342-only scores are not converted into this bank. Not ranked here. Checked / reported: 2026-11-02 · Samples: 50 Snapshot: 2026-11-01 · Revision: f5fa9cfafccd97fcfbdb0336e14284faaa8d2d35 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
System 1 Decision Benchmark v2 · F19 · Exceptions and priorities	98.276%	Setup & source Exceptions and priorities: 58 questions. Mammoth-1 (Baseten direct; latest named alias); Hosted native; returned identities {"decision-models/baseten-mammoth-27b-latest": 1933, "unreported in selected primary": 1}; collected 2026-10-21 to 2026-10-31 America/Toronto; one primary response per input, no scored retries. Unsuccessful responses counted as wrong: {"service/rate limit": 1}. Development benchmark, not an independently sampled test: 1,934 questions in 41 categories, the frozen 1,342-question bank plus 192 logical and 400 practical additions. The bank was built with model-guided development, and Jev plus retained Microsoft and Drex feedback are development exposure. Related variants and a single recorded primary response per input limit generalization. The scoring key stays private, so public checks cover coverage, timing and aggregate consistency but cannot rescore answers. Strict accuracy counts a wrong usable answer and an unsuccessful response in the same fixed denominator. Timing pools primary client times with the network included; providers, precision, concurrency, routes and dates differ, retained and new timings can span sessions, and quick failures can reduce all-primary latency. The source published a separate cost ledger on 1 November 2026, stored as decision-reference-v2-cost-per-1000, which supersedes its earlier “cost not consolidated” note. Older 1,342-only scores are not converted into this bank. Not ranked here. Checked / reported: 2026-11-02 · Samples: 58 Snapshot: 2026-11-01 · Revision: f5fa9cfafccd97fcfbdb0336e14284faaa8d2d35 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
System 1 Decision Benchmark v2 · F20 · Planning	90.698%	Setup & source Planning: 43 questions. Mammoth-1 (Baseten direct; latest named alias); Hosted native; returned identities {"decision-models/baseten-mammoth-27b-latest": 1933, "unreported in selected primary": 1}; collected 2026-10-21 to 2026-10-31 America/Toronto; one primary response per input, no scored retries. Unsuccessful responses counted as wrong: {"service/rate limit": 1}. Development benchmark, not an independently sampled test: 1,934 questions in 41 categories, the frozen 1,342-question bank plus 192 logical and 400 practical additions. The bank was built with model-guided development, and Jev plus retained Microsoft and Drex feedback are development exposure. Related variants and a single recorded primary response per input limit generalization. The scoring key stays private, so public checks cover coverage, timing and aggregate consistency but cannot rescore answers. Strict accuracy counts a wrong usable answer and an unsuccessful response in the same fixed denominator. Timing pools primary client times with the network included; providers, precision, concurrency, routes and dates differ, retained and new timings can span sessions, and quick failures can reduce all-primary latency. The source published a separate cost ledger on 1 November 2026, stored as decision-reference-v2-cost-per-1000, which supersedes its earlier “cost not consolidated” note. Older 1,342-only scores are not converted into this bank. Not ranked here. Checked / reported: 2026-11-02 · Samples: 43 Snapshot: 2026-11-01 · Revision: f5fa9cfafccd97fcfbdb0336e14284faaa8d2d35 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
System 1 Decision Benchmark v2 · F21 · Directed reachability	95.349%	Setup & source Directed reachability: 43 questions. Mammoth-1 (Baseten direct; latest named alias); Hosted native; returned identities {"decision-models/baseten-mammoth-27b-latest": 1933, "unreported in selected primary": 1}; collected 2026-10-21 to 2026-10-31 America/Toronto; one primary response per input, no scored retries. Unsuccessful responses counted as wrong: {"service/rate limit": 1}. Development benchmark, not an independently sampled test: 1,934 questions in 41 categories, the frozen 1,342-question bank plus 192 logical and 400 practical additions. The bank was built with model-guided development, and Jev plus retained Microsoft and Drex feedback are development exposure. Related variants and a single recorded primary response per input limit generalization. The scoring key stays private, so public checks cover coverage, timing and aggregate consistency but cannot rescore answers. Strict accuracy counts a wrong usable answer and an unsuccessful response in the same fixed denominator. Timing pools primary client times with the network included; providers, precision, concurrency, routes and dates differ, retained and new timings can span sessions, and quick failures can reduce all-primary latency. The source published a separate cost ledger on 1 November 2026, stored as decision-reference-v2-cost-per-1000, which supersedes its earlier “cost not consolidated” note. Older 1,342-only scores are not converted into this bank. Not ranked here. Checked / reported: 2026-11-02 · Samples: 43 Snapshot: 2026-11-01 · Revision: f5fa9cfafccd97fcfbdb0336e14284faaa8d2d35 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
System 1 Decision Benchmark v2 · F22 · Preference ordering	94.118%	Setup & source Preference ordering: 51 questions. Mammoth-1 (Baseten direct; latest named alias); Hosted native; returned identities {"decision-models/baseten-mammoth-27b-latest": 1933, "unreported in selected primary": 1}; collected 2026-10-21 to 2026-10-31 America/Toronto; one primary response per input, no scored retries. Unsuccessful responses counted as wrong: {"service/rate limit": 1}. Development benchmark, not an independently sampled test: 1,934 questions in 41 categories, the frozen 1,342-question bank plus 192 logical and 400 practical additions. The bank was built with model-guided development, and Jev plus retained Microsoft and Drex feedback are development exposure. Related variants and a single recorded primary response per input limit generalization. The scoring key stays private, so public checks cover coverage, timing and aggregate consistency but cannot rescore answers. Strict accuracy counts a wrong usable answer and an unsuccessful response in the same fixed denominator. Timing pools primary client times with the network included; providers, precision, concurrency, routes and dates differ, retained and new timings can span sessions, and quick failures can reduce all-primary latency. The source published a separate cost ledger on 1 November 2026, stored as decision-reference-v2-cost-per-1000, which supersedes its earlier “cost not consolidated” note. Older 1,342-only scores are not converted into this bank. Not ranked here. Checked / reported: 2026-11-02 · Samples: 51 Snapshot: 2026-11-01 · Revision: f5fa9cfafccd97fcfbdb0336e14284faaa8d2d35 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
System 1 Decision Benchmark v2 · F23 · Rule counterexamples	100.000%	Setup & source Rule counterexamples: 52 questions. Mammoth-1 (Baseten direct; latest named alias); Hosted native; returned identities {"decision-models/baseten-mammoth-27b-latest": 1933, "unreported in selected primary": 1}; collected 2026-10-21 to 2026-10-31 America/Toronto; one primary response per input, no scored retries. Unsuccessful responses counted as wrong: {"service/rate limit": 1}. Development benchmark, not an independently sampled test: 1,934 questions in 41 categories, the frozen 1,342-question bank plus 192 logical and 400 practical additions. The bank was built with model-guided development, and Jev plus retained Microsoft and Drex feedback are development exposure. Related variants and a single recorded primary response per input limit generalization. The scoring key stays private, so public checks cover coverage, timing and aggregate consistency but cannot rescore answers. Strict accuracy counts a wrong usable answer and an unsuccessful response in the same fixed denominator. Timing pools primary client times with the network included; providers, precision, concurrency, routes and dates differ, retained and new timings can span sessions, and quick failures can reduce all-primary latency. The source published a separate cost ledger on 1 November 2026, stored as decision-reference-v2-cost-per-1000, which supersedes its earlier “cost not consolidated” note. Older 1,342-only scores are not converted into this bank. Not ranked here. Checked / reported: 2026-11-02 · Samples: 52 Snapshot: 2026-11-01 · Revision: f5fa9cfafccd97fcfbdb0336e14284faaa8d2d35 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
System 1 Decision Benchmark v2 · F24 · Authority and stale records	97.674%	Setup & source Authority and stale records: 43 questions. Mammoth-1 (Baseten direct; latest named alias); Hosted native; returned identities {"decision-models/baseten-mammoth-27b-latest": 1933, "unreported in selected primary": 1}; collected 2026-10-21 to 2026-10-31 America/Toronto; one primary response per input, no scored retries. Unsuccessful responses counted as wrong: {"service/rate limit": 1}. Development benchmark, not an independently sampled test: 1,934 questions in 41 categories, the frozen 1,342-question bank plus 192 logical and 400 practical additions. The bank was built with model-guided development, and Jev plus retained Microsoft and Drex feedback are development exposure. Related variants and a single recorded primary response per input limit generalization. The scoring key stays private, so public checks cover coverage, timing and aggregate consistency but cannot rescore answers. Strict accuracy counts a wrong usable answer and an unsuccessful response in the same fixed denominator. Timing pools primary client times with the network included; providers, precision, concurrency, routes and dates differ, retained and new timings can span sessions, and quick failures can reduce all-primary latency. The source published a separate cost ledger on 1 November 2026, stored as decision-reference-v2-cost-per-1000, which supersedes its earlier “cost not consolidated” note. Older 1,342-only scores are not converted into this bank. Not ranked here. Checked / reported: 2026-11-02 · Samples: 43 Snapshot: 2026-11-01 · Revision: f5fa9cfafccd97fcfbdb0336e14284faaa8d2d35 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
System 1 Decision Benchmark v2 · F25 · Fault diagnosis	94.000%	Setup & source Fault diagnosis: 50 questions. Mammoth-1 (Baseten direct; latest named alias); Hosted native; returned identities {"decision-models/baseten-mammoth-27b-latest": 1933, "unreported in selected primary": 1}; collected 2026-10-21 to 2026-10-31 America/Toronto; one primary response per input, no scored retries. Unsuccessful responses counted as wrong: {"service/rate limit": 1}. Development benchmark, not an independently sampled test: 1,934 questions in 41 categories, the frozen 1,342-question bank plus 192 logical and 400 practical additions. The bank was built with model-guided development, and Jev plus retained Microsoft and Drex feedback are development exposure. Related variants and a single recorded primary response per input limit generalization. The scoring key stays private, so public checks cover coverage, timing and aggregate consistency but cannot rescore answers. Strict accuracy counts a wrong usable answer and an unsuccessful response in the same fixed denominator. Timing pools primary client times with the network included; providers, precision, concurrency, routes and dates differ, retained and new timings can span sessions, and quick failures can reduce all-primary latency. The source published a separate cost ledger on 1 November 2026, stored as decision-reference-v2-cost-per-1000, which supersedes its earlier “cost not consolidated” note. Older 1,342-only scores are not converted into this bank. Not ranked here. Checked / reported: 2026-11-02 · Samples: 50 Snapshot: 2026-11-01 · Revision: f5fa9cfafccd97fcfbdb0336e14284faaa8d2d35 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
System 1 Decision Benchmark v2 · F26 · Probabilistic risk	70.000%	Setup & source Probabilistic risk: 40 questions. Mammoth-1 (Baseten direct; latest named alias); Hosted native; returned identities {"decision-models/baseten-mammoth-27b-latest": 1933, "unreported in selected primary": 1}; collected 2026-10-21 to 2026-10-31 America/Toronto; one primary response per input, no scored retries. Unsuccessful responses counted as wrong: {"service/rate limit": 1}. Development benchmark, not an independently sampled test: 1,934 questions in 41 categories, the frozen 1,342-question bank plus 192 logical and 400 practical additions. The bank was built with model-guided development, and Jev plus retained Microsoft and Drex feedback are development exposure. Related variants and a single recorded primary response per input limit generalization. The scoring key stays private, so public checks cover coverage, timing and aggregate consistency but cannot rescore answers. Strict accuracy counts a wrong usable answer and an unsuccessful response in the same fixed denominator. Timing pools primary client times with the network included; providers, precision, concurrency, routes and dates differ, retained and new timings can span sessions, and quick failures can reduce all-primary latency. The source published a separate cost ledger on 1 November 2026, stored as decision-reference-v2-cost-per-1000, which supersedes its earlier “cost not consolidated” note. Older 1,342-only scores are not converted into this bank. Not ranked here. Checked / reported: 2026-11-02 · Samples: 40 Snapshot: 2026-11-01 · Revision: f5fa9cfafccd97fcfbdb0336e14284faaa8d2d35 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
System 1 Decision Benchmark v2 · F27 · Value of information	65.000%	Setup & source Value of information: 40 questions. Mammoth-1 (Baseten direct; latest named alias); Hosted native; returned identities {"decision-models/baseten-mammoth-27b-latest": 1933, "unreported in selected primary": 1}; collected 2026-10-21 to 2026-10-31 America/Toronto; one primary response per input, no scored retries. Unsuccessful responses counted as wrong: {"service/rate limit": 1}. Development benchmark, not an independently sampled test: 1,934 questions in 41 categories, the frozen 1,342-question bank plus 192 logical and 400 practical additions. The bank was built with model-guided development, and Jev plus retained Microsoft and Drex feedback are development exposure. Related variants and a single recorded primary response per input limit generalization. The scoring key stays private, so public checks cover coverage, timing and aggregate consistency but cannot rescore answers. Strict accuracy counts a wrong usable answer and an unsuccessful response in the same fixed denominator. Timing pools primary client times with the network included; providers, precision, concurrency, routes and dates differ, retained and new timings can span sessions, and quick failures can reduce all-primary latency. The source published a separate cost ledger on 1 November 2026, stored as decision-reference-v2-cost-per-1000, which supersedes its earlier “cost not consolidated” note. Older 1,342-only scores are not converted into this bank. Not ranked here. Checked / reported: 2026-11-02 · Samples: 40 Snapshot: 2026-11-01 · Revision: f5fa9cfafccd97fcfbdb0336e14284faaa8d2d35 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
System 1 Decision Benchmark v2 · F28 · Robust decisions without a prior	25.000%	Setup & source Robust decisions without a prior: 40 questions. Mammoth-1 (Baseten direct; latest named alias); Hosted native; returned identities {"decision-models/baseten-mammoth-27b-latest": 1933, "unreported in selected primary": 1}; collected 2026-10-21 to 2026-10-31 America/Toronto; one primary response per input, no scored retries. Unsuccessful responses counted as wrong: {"service/rate limit": 1}. Development benchmark, not an independently sampled test: 1,934 questions in 41 categories, the frozen 1,342-question bank plus 192 logical and 400 practical additions. The bank was built with model-guided development, and Jev plus retained Microsoft and Drex feedback are development exposure. Related variants and a single recorded primary response per input limit generalization. The scoring key stays private, so public checks cover coverage, timing and aggregate consistency but cannot rescore answers. Strict accuracy counts a wrong usable answer and an unsuccessful response in the same fixed denominator. Timing pools primary client times with the network included; providers, precision, concurrency, routes and dates differ, retained and new timings can span sessions, and quick failures can reduce all-primary latency. The source published a separate cost ledger on 1 November 2026, stored as decision-reference-v2-cost-per-1000, which supersedes its earlier “cost not consolidated” note. Older 1,342-only scores are not converted into this bank. Not ranked here. Checked / reported: 2026-11-02 · Samples: 40 Snapshot: 2026-11-01 · Revision: f5fa9cfafccd97fcfbdb0336e14284faaa8d2d35 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
System 1 Decision Benchmark v2 · F29 · Strategic decisions	77.500%	Setup & source Strategic decisions: 40 questions. Mammoth-1 (Baseten direct; latest named alias); Hosted native; returned identities {"decision-models/baseten-mammoth-27b-latest": 1933, "unreported in selected primary": 1}; collected 2026-10-21 to 2026-10-31 America/Toronto; one primary response per input, no scored retries. Unsuccessful responses counted as wrong: {"service/rate limit": 1}. Development benchmark, not an independently sampled test: 1,934 questions in 41 categories, the frozen 1,342-question bank plus 192 logical and 400 practical additions. The bank was built with model-guided development, and Jev plus retained Microsoft and Drex feedback are development exposure. Related variants and a single recorded primary response per input limit generalization. The scoring key stays private, so public checks cover coverage, timing and aggregate consistency but cannot rescore answers. Strict accuracy counts a wrong usable answer and an unsuccessful response in the same fixed denominator. Timing pools primary client times with the network included; providers, precision, concurrency, routes and dates differ, retained and new timings can span sessions, and quick failures can reduce all-primary latency. The source published a separate cost ledger on 1 November 2026, stored as decision-reference-v2-cost-per-1000, which supersedes its earlier “cost not consolidated” note. Older 1,342-only scores are not converted into this bank. Not ranked here. Checked / reported: 2026-11-02 · Samples: 40 Snapshot: 2026-11-01 · Revision: f5fa9cfafccd97fcfbdb0336e14284faaa8d2d35 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
System 1 Decision Benchmark v2 · F30 · Concurrent-history consistency	60.000%	Setup & source Concurrent-history consistency: 40 questions. Mammoth-1 (Baseten direct; latest named alias); Hosted native; returned identities {"decision-models/baseten-mammoth-27b-latest": 1933, "unreported in selected primary": 1}; collected 2026-10-21 to 2026-10-31 America/Toronto; one primary response per input, no scored retries. Unsuccessful responses counted as wrong: {"service/rate limit": 1}. Development benchmark, not an independently sampled test: 1,934 questions in 41 categories, the frozen 1,342-question bank plus 192 logical and 400 practical additions. The bank was built with model-guided development, and Jev plus retained Microsoft and Drex feedback are development exposure. Related variants and a single recorded primary response per input limit generalization. The scoring key stays private, so public checks cover coverage, timing and aggregate consistency but cannot rescore answers. Strict accuracy counts a wrong usable answer and an unsuccessful response in the same fixed denominator. Timing pools primary client times with the network included; providers, precision, concurrency, routes and dates differ, retained and new timings can span sessions, and quick failures can reduce all-primary latency. The source published a separate cost ledger on 1 November 2026, stored as decision-reference-v2-cost-per-1000, which supersedes its earlier “cost not consolidated” note. Older 1,342-only scores are not converted into this bank. Not ranked here. Checked / reported: 2026-11-02 · Samples: 40 Snapshot: 2026-11-01 · Revision: f5fa9cfafccd97fcfbdb0336e14284faaa8d2d35 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
System 1 Decision Benchmark v2 · F31 · Information-flow security	77.500%	Setup & source Information-flow security: 40 questions. Mammoth-1 (Baseten direct; latest named alias); Hosted native; returned identities {"decision-models/baseten-mammoth-27b-latest": 1933, "unreported in selected primary": 1}; collected 2026-10-21 to 2026-10-31 America/Toronto; one primary response per input, no scored retries. Unsuccessful responses counted as wrong: {"service/rate limit": 1}. Development benchmark, not an independently sampled test: 1,934 questions in 41 categories, the frozen 1,342-question bank plus 192 logical and 400 practical additions. The bank was built with model-guided development, and Jev plus retained Microsoft and Drex feedback are development exposure. Related variants and a single recorded primary response per input limit generalization. The scoring key stays private, so public checks cover coverage, timing and aggregate consistency but cannot rescore answers. Strict accuracy counts a wrong usable answer and an unsuccessful response in the same fixed denominator. Timing pools primary client times with the network included; providers, precision, concurrency, routes and dates differ, retained and new timings can span sessions, and quick failures can reduce all-primary latency. The source published a separate cost ledger on 1 November 2026, stored as decision-reference-v2-cost-per-1000, which supersedes its earlier “cost not consolidated” note. Older 1,342-only scores are not converted into this bank. Not ranked here. Checked / reported: 2026-11-02 · Samples: 40 Snapshot: 2026-11-01 · Revision: f5fa9cfafccd97fcfbdb0336e14284faaa8d2d35 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
System 1 Decision Benchmark v2 · P01 · Understanding requests and routing	95.000%	Setup & source Understanding requests and routing: 40 questions. Mammoth-1 (Baseten direct; latest named alias); Hosted native; returned identities {"decision-models/baseten-mammoth-27b-latest": 1933, "unreported in selected primary": 1}; collected 2026-10-21 to 2026-10-31 America/Toronto; one primary response per input, no scored retries. Unsuccessful responses counted as wrong: {"service/rate limit": 1}. Development benchmark, not an independently sampled test: 1,934 questions in 41 categories, the frozen 1,342-question bank plus 192 logical and 400 practical additions. The bank was built with model-guided development, and Jev plus retained Microsoft and Drex feedback are development exposure. Related variants and a single recorded primary response per input limit generalization. The scoring key stays private, so public checks cover coverage, timing and aggregate consistency but cannot rescore answers. Strict accuracy counts a wrong usable answer and an unsuccessful response in the same fixed denominator. Timing pools primary client times with the network included; providers, precision, concurrency, routes and dates differ, retained and new timings can span sessions, and quick failures can reduce all-primary latency. The source published a separate cost ledger on 1 November 2026, stored as decision-reference-v2-cost-per-1000, which supersedes its earlier “cost not consolidated” note. Older 1,342-only scores are not converted into this bank. Not ranked here. Checked / reported: 2026-11-02 · Samples: 40 Snapshot: 2026-11-01 · Revision: f5fa9cfafccd97fcfbdb0336e14284faaa8d2d35 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
System 1 Decision Benchmark v2 · P02 · Recognizing multiple requested actions	80.000%	Setup & source Recognizing multiple requested actions: 40 questions. Mammoth-1 (Baseten direct; latest named alias); Hosted native; returned identities {"decision-models/baseten-mammoth-27b-latest": 1933, "unreported in selected primary": 1}; collected 2026-10-21 to 2026-10-31 America/Toronto; one primary response per input, no scored retries. Unsuccessful responses counted as wrong: {"service/rate limit": 1}. Development benchmark, not an independently sampled test: 1,934 questions in 41 categories, the frozen 1,342-question bank plus 192 logical and 400 practical additions. The bank was built with model-guided development, and Jev plus retained Microsoft and Drex feedback are development exposure. Related variants and a single recorded primary response per input limit generalization. The scoring key stays private, so public checks cover coverage, timing and aggregate consistency but cannot rescore answers. Strict accuracy counts a wrong usable answer and an unsuccessful response in the same fixed denominator. Timing pools primary client times with the network included; providers, precision, concurrency, routes and dates differ, retained and new timings can span sessions, and quick failures can reduce all-primary latency. The source published a separate cost ledger on 1 November 2026, stored as decision-reference-v2-cost-per-1000, which supersedes its earlier “cost not consolidated” note. Older 1,342-only scores are not converted into this bank. Not ranked here. Checked / reported: 2026-11-02 · Samples: 40 Snapshot: 2026-11-01 · Revision: f5fa9cfafccd97fcfbdb0336e14284faaa8d2d35 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
System 1 Decision Benchmark v2 · P03 · Tracking current conversational intention	100.000%	Setup & source Tracking current conversational intention: 40 questions. Mammoth-1 (Baseten direct; latest named alias); Hosted native; returned identities {"decision-models/baseten-mammoth-27b-latest": 1933, "unreported in selected primary": 1}; collected 2026-10-21 to 2026-10-31 America/Toronto; one primary response per input, no scored retries. Unsuccessful responses counted as wrong: {"service/rate limit": 1}. Development benchmark, not an independently sampled test: 1,934 questions in 41 categories, the frozen 1,342-question bank plus 192 logical and 400 practical additions. The bank was built with model-guided development, and Jev plus retained Microsoft and Drex feedback are development exposure. Related variants and a single recorded primary response per input limit generalization. The scoring key stays private, so public checks cover coverage, timing and aggregate consistency but cannot rescore answers. Strict accuracy counts a wrong usable answer and an unsuccessful response in the same fixed denominator. Timing pools primary client times with the network included; providers, precision, concurrency, routes and dates differ, retained and new timings can span sessions, and quick failures can reduce all-primary latency. The source published a separate cost ledger on 1 November 2026, stored as decision-reference-v2-cost-per-1000, which supersedes its earlier “cost not consolidated” note. Older 1,342-only scores are not converted into this bank. Not ranked here. Checked / reported: 2026-11-02 · Samples: 40 Snapshot: 2026-11-01 · Revision: f5fa9cfafccd97fcfbdb0336e14284faaa8d2d35 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
System 1 Decision Benchmark v2 · P04 · Selecting a tool or requesting clarification	57.500%	Setup & source Selecting a tool or requesting clarification: 40 questions. Mammoth-1 (Baseten direct; latest named alias); Hosted native; returned identities {"decision-models/baseten-mammoth-27b-latest": 1933, "unreported in selected primary": 1}; collected 2026-10-21 to 2026-10-31 America/Toronto; one primary response per input, no scored retries. Unsuccessful responses counted as wrong: {"service/rate limit": 1}. Development benchmark, not an independently sampled test: 1,934 questions in 41 categories, the frozen 1,342-question bank plus 192 logical and 400 practical additions. The bank was built with model-guided development, and Jev plus retained Microsoft and Drex feedback are development exposure. Related variants and a single recorded primary response per input limit generalization. The scoring key stays private, so public checks cover coverage, timing and aggregate consistency but cannot rescore answers. Strict accuracy counts a wrong usable answer and an unsuccessful response in the same fixed denominator. Timing pools primary client times with the network included; providers, precision, concurrency, routes and dates differ, retained and new timings can span sessions, and quick failures can reduce all-primary latency. The source published a separate cost ledger on 1 November 2026, stored as decision-reference-v2-cost-per-1000, which supersedes its earlier “cost not consolidated” note. Older 1,342-only scores are not converted into this bank. Not ranked here. Checked / reported: 2026-11-02 · Samples: 40 Snapshot: 2026-11-01 · Revision: f5fa9cfafccd97fcfbdb0336e14284faaa8d2d35 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
System 1 Decision Benchmark v2 · P05 · Checking authorization and scope	52.500%	Setup & source Checking authorization and scope: 40 questions. Mammoth-1 (Baseten direct; latest named alias); Hosted native; returned identities {"decision-models/baseten-mammoth-27b-latest": 1933, "unreported in selected primary": 1}; collected 2026-10-21 to 2026-10-31 America/Toronto; one primary response per input, no scored retries. Unsuccessful responses counted as wrong: {"service/rate limit": 1}. Development benchmark, not an independently sampled test: 1,934 questions in 41 categories, the frozen 1,342-question bank plus 192 logical and 400 practical additions. The bank was built with model-guided development, and Jev plus retained Microsoft and Drex feedback are development exposure. Related variants and a single recorded primary response per input limit generalization. The scoring key stays private, so public checks cover coverage, timing and aggregate consistency but cannot rescore answers. Strict accuracy counts a wrong usable answer and an unsuccessful response in the same fixed denominator. Timing pools primary client times with the network included; providers, precision, concurrency, routes and dates differ, retained and new timings can span sessions, and quick failures can reduce all-primary latency. The source published a separate cost ledger on 1 November 2026, stored as decision-reference-v2-cost-per-1000, which supersedes its earlier “cost not consolidated” note. Older 1,342-only scores are not converted into this bank. Not ranked here. Checked / reported: 2026-11-02 · Samples: 40 Snapshot: 2026-11-01 · Revision: f5fa9cfafccd97fcfbdb0336e14284faaa8d2d35 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
System 1 Decision Benchmark v2 · P06 · Making decisions with untrusted content	90.000%	Setup & source Making decisions with untrusted content: 40 questions. Mammoth-1 (Baseten direct; latest named alias); Hosted native; returned identities {"decision-models/baseten-mammoth-27b-latest": 1933, "unreported in selected primary": 1}; collected 2026-10-21 to 2026-10-31 America/Toronto; one primary response per input, no scored retries. Unsuccessful responses counted as wrong: {"service/rate limit": 1}. Development benchmark, not an independently sampled test: 1,934 questions in 41 categories, the frozen 1,342-question bank plus 192 logical and 400 practical additions. The bank was built with model-guided development, and Jev plus retained Microsoft and Drex feedback are development exposure. Related variants and a single recorded primary response per input limit generalization. The scoring key stays private, so public checks cover coverage, timing and aggregate consistency but cannot rescore answers. Strict accuracy counts a wrong usable answer and an unsuccessful response in the same fixed denominator. Timing pools primary client times with the network included; providers, precision, concurrency, routes and dates differ, retained and new timings can span sessions, and quick failures can reduce all-primary latency. The source published a separate cost ledger on 1 November 2026, stored as decision-reference-v2-cost-per-1000, which supersedes its earlier “cost not consolidated” note. Older 1,342-only scores are not converted into this bank. Not ranked here. Checked / reported: 2026-11-02 · Samples: 40 Snapshot: 2026-11-01 · Revision: f5fa9cfafccd97fcfbdb0336e14284faaa8d2d35 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
System 1 Decision Benchmark v2 · P07 · Applying a contextual content policy	72.500%	Setup & source Applying a contextual content policy: 40 questions. Mammoth-1 (Baseten direct; latest named alias); Hosted native; returned identities {"decision-models/baseten-mammoth-27b-latest": 1933, "unreported in selected primary": 1}; collected 2026-10-21 to 2026-10-31 America/Toronto; one primary response per input, no scored retries. Unsuccessful responses counted as wrong: {"service/rate limit": 1}. Development benchmark, not an independently sampled test: 1,934 questions in 41 categories, the frozen 1,342-question bank plus 192 logical and 400 practical additions. The bank was built with model-guided development, and Jev plus retained Microsoft and Drex feedback are development exposure. Related variants and a single recorded primary response per input limit generalization. The scoring key stays private, so public checks cover coverage, timing and aggregate consistency but cannot rescore answers. Strict accuracy counts a wrong usable answer and an unsuccessful response in the same fixed denominator. Timing pools primary client times with the network included; providers, precision, concurrency, routes and dates differ, retained and new timings can span sessions, and quick failures can reduce all-primary latency. The source published a separate cost ledger on 1 November 2026, stored as decision-reference-v2-cost-per-1000, which supersedes its earlier “cost not consolidated” note. Older 1,342-only scores are not converted into this bank. Not ranked here. Checked / reported: 2026-11-02 · Samples: 40 Snapshot: 2026-11-01 · Revision: f5fa9cfafccd97fcfbdb0336e14284faaa8d2d35 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
System 1 Decision Benchmark v2 · P08 · Extracting and normalizing facts	52.500%	Setup & source Extracting and normalizing facts: 40 questions. Mammoth-1 (Baseten direct; latest named alias); Hosted native; returned identities {"decision-models/baseten-mammoth-27b-latest": 1933, "unreported in selected primary": 1}; collected 2026-10-21 to 2026-10-31 America/Toronto; one primary response per input, no scored retries. Unsuccessful responses counted as wrong: {"service/rate limit": 1}. Development benchmark, not an independently sampled test: 1,934 questions in 41 categories, the frozen 1,342-question bank plus 192 logical and 400 practical additions. The bank was built with model-guided development, and Jev plus retained Microsoft and Drex feedback are development exposure. Related variants and a single recorded primary response per input limit generalization. The scoring key stays private, so public checks cover coverage, timing and aggregate consistency but cannot rescore answers. Strict accuracy counts a wrong usable answer and an unsuccessful response in the same fixed denominator. Timing pools primary client times with the network included; providers, precision, concurrency, routes and dates differ, retained and new timings can span sessions, and quick failures can reduce all-primary latency. The source published a separate cost ledger on 1 November 2026, stored as decision-reference-v2-cost-per-1000, which supersedes its earlier “cost not consolidated” note. Older 1,342-only scores are not converted into this bank. Not ranked here. Checked / reported: 2026-11-02 · Samples: 40 Snapshot: 2026-11-01 · Revision: f5fa9cfafccd97fcfbdb0336e14284faaa8d2d35 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
System 1 Decision Benchmark v2 · P09 · Checking claims against supplied evidence	77.500%	Setup & source Checking claims against supplied evidence: 40 questions. Mammoth-1 (Baseten direct; latest named alias); Hosted native; returned identities {"decision-models/baseten-mammoth-27b-latest": 1933, "unreported in selected primary": 1}; collected 2026-10-21 to 2026-10-31 America/Toronto; one primary response per input, no scored retries. Unsuccessful responses counted as wrong: {"service/rate limit": 1}. Development benchmark, not an independently sampled test: 1,934 questions in 41 categories, the frozen 1,342-question bank plus 192 logical and 400 practical additions. The bank was built with model-guided development, and Jev plus retained Microsoft and Drex feedback are development exposure. Related variants and a single recorded primary response per input limit generalization. The scoring key stays private, so public checks cover coverage, timing and aggregate consistency but cannot rescore answers. Strict accuracy counts a wrong usable answer and an unsuccessful response in the same fixed denominator. Timing pools primary client times with the network included; providers, precision, concurrency, routes and dates differ, retained and new timings can span sessions, and quick failures can reduce all-primary latency. The source published a separate cost ledger on 1 November 2026, stored as decision-reference-v2-cost-per-1000, which supersedes its earlier “cost not consolidated” note. Older 1,342-only scores are not converted into this bank. Not ranked here. Checked / reported: 2026-11-02 · Samples: 40 Snapshot: 2026-11-01 · Revision: f5fa9cfafccd97fcfbdb0336e14284faaa8d2d35 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
System 1 Decision Benchmark v2 · P10 · Assigning severity or priority from a rubric	75.000%	Setup & source Assigning severity or priority from a rubric: 40 questions. Mammoth-1 (Baseten direct; latest named alias); Hosted native; returned identities {"decision-models/baseten-mammoth-27b-latest": 1933, "unreported in selected primary": 1}; collected 2026-10-21 to 2026-10-31 America/Toronto; one primary response per input, no scored retries. Unsuccessful responses counted as wrong: {"service/rate limit": 1}. Development benchmark, not an independently sampled test: 1,934 questions in 41 categories, the frozen 1,342-question bank plus 192 logical and 400 practical additions. The bank was built with model-guided development, and Jev plus retained Microsoft and Drex feedback are development exposure. Related variants and a single recorded primary response per input limit generalization. The scoring key stays private, so public checks cover coverage, timing and aggregate consistency but cannot rescore answers. Strict accuracy counts a wrong usable answer and an unsuccessful response in the same fixed denominator. Timing pools primary client times with the network included; providers, precision, concurrency, routes and dates differ, retained and new timings can span sessions, and quick failures can reduce all-primary latency. The source published a separate cost ledger on 1 November 2026, stored as decision-reference-v2-cost-per-1000, which supersedes its earlier “cost not consolidated” note. Older 1,342-only scores are not converted into this bank. Not ranked here. Checked / reported: 2026-11-02 · Samples: 40 Snapshot: 2026-11-01 · Revision: f5fa9cfafccd97fcfbdb0336e14284faaa8d2d35 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
System 1 Decision Benchmark v2 · Tariff-based cost per 1,000 questions	0.02733 USD/1k decisions	Setup & source Captured-tariff estimate for primary calls without a reported charge, in USD per 1,000 questions. Mammoth-1 (latest named alias); System 1 v2 route; evidence captured_tariff_or_mixed. 591 calls at a captured tariff, 1342 on free access, 1 unknown, out of 1,934. Tariff-based estimate from the same ledger, kept apart from the reported cost dataset. These calls have no provider-reported charge: the ledger applied a captured tariff, and calls on free access during a launch week add nothing, so a value can understate the paid rate. Providers and routes differ; these are not current tariffs. Development benchmark; not ranked here. Checked / reported: 2026-11-02 · Samples: 1934 Snapshot: 2026-11-02 · Revision: 5b47806ba527db5e1abc88578789f3eae6e8e646 · Published summary observations; upstream task/result terms retained Result source · Benchmark source

Independent · ModelTested

mammoth-1 · Independent · ModelTested
Evaluation	Result	Evidence
ModelTested 56-case · All-case accuracy	92.00%	Setup & source Exact named hosted API observation; All-case accuracy. Named source route baseten/mammoth-1; ModelTested runner v1; tested_on=openrouter. Backend weight revision and API inference precision unpublished. Publisher generated 2026-11-01; board added this route 2026-10-29. Execution dates, token usage and hashes remain in its per-run receipts. No private prompt/response archive imported. 56 authored situations, including 19 hard cases, each with two typed questions; 14 public and 42 hidden cases. Source aggregate averages per-case decision accuracy. Confidence score is 100×(1−mean multiclass Brier/2), not ECE or a calibration probability. Small single-run benchmark; current HF access/size does not inherit this evaluator’s stale closed/API-only metadata. No tariff inferred from rounded cost 0.0. Rows added 1 November 2026 use revision 668a52ad3b932ca910fd985f96dcdf0a85d2dae1, where the earlier 11 systems are unchanged. Rows added 2 November 2026 use revision 5c6c19bb3b7332ccd28c8c13bd32cdc69cedc8f0. Checked / reported: 2026-11-01 · Samples: 56 Snapshot: 2026-10-18 · Revision: 081e8f37130a4a9077b8cffe62827d4c3b9e3b23; 668a52ad3b932ca910fd985f96dcdf0a85d2dae1 for Drex Max, Jev 1.14 and d1.1; 5c6c19bb3b7332ccd28c8c13bd32cdc69cedc8f0 for Mammoth, Elephant, Joey, Span-02 and both Solar Decide routes · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
ModelTested 56-case · Hard-case accuracy	86.80%	Setup & source Exact named hosted API observation; Hard-case accuracy. Named source route baseten/mammoth-1; ModelTested runner v1; tested_on=openrouter. Backend weight revision and API inference precision unpublished. Publisher generated 2026-11-01; board added this route 2026-10-29. Execution dates, token usage and hashes remain in its per-run receipts. No private prompt/response archive imported. 56 authored situations, including 19 hard cases, each with two typed questions; 14 public and 42 hidden cases. Source aggregate averages per-case decision accuracy. Confidence score is 100×(1−mean multiclass Brier/2), not ECE or a calibration probability. Small single-run benchmark; current HF access/size does not inherit this evaluator’s stale closed/API-only metadata. No tariff inferred from rounded cost 0.0. Rows added 1 November 2026 use revision 668a52ad3b932ca910fd985f96dcdf0a85d2dae1, where the earlier 11 systems are unchanged. Rows added 2 November 2026 use revision 5c6c19bb3b7332ccd28c8c13bd32cdc69cedc8f0. Checked / reported: 2026-11-01 · Samples: 56 Snapshot: 2026-10-18 · Revision: 081e8f37130a4a9077b8cffe62827d4c3b9e3b23; 668a52ad3b932ca910fd985f96dcdf0a85d2dae1 for Drex Max, Jev 1.14 and d1.1; 5c6c19bb3b7332ccd28c8c13bd32cdc69cedc8f0 for Mammoth, Elephant, Joey, Span-02 and both Solar Decide routes · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
ModelTested 56-case · Public-case accuracy	96.40%	Setup & source Exact named hosted API observation; Public-case accuracy. Named source route baseten/mammoth-1; ModelTested runner v1; tested_on=openrouter. Backend weight revision and API inference precision unpublished. Publisher generated 2026-11-01; board added this route 2026-10-29. Execution dates, token usage and hashes remain in its per-run receipts. No private prompt/response archive imported. 56 authored situations, including 19 hard cases, each with two typed questions; 14 public and 42 hidden cases. Source aggregate averages per-case decision accuracy. Confidence score is 100×(1−mean multiclass Brier/2), not ECE or a calibration probability. Small single-run benchmark; current HF access/size does not inherit this evaluator’s stale closed/API-only metadata. No tariff inferred from rounded cost 0.0. Rows added 1 November 2026 use revision 668a52ad3b932ca910fd985f96dcdf0a85d2dae1, where the earlier 11 systems are unchanged. Rows added 2 November 2026 use revision 5c6c19bb3b7332ccd28c8c13bd32cdc69cedc8f0. Checked / reported: 2026-11-01 · Samples: 56 Snapshot: 2026-10-18 · Revision: 081e8f37130a4a9077b8cffe62827d4c3b9e3b23; 668a52ad3b932ca910fd985f96dcdf0a85d2dae1 for Drex Max, Jev 1.14 and d1.1; 5c6c19bb3b7332ccd28c8c13bd32cdc69cedc8f0 for Mammoth, Elephant, Joey, Span-02 and both Solar Decide routes · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
ModelTested 56-case · Hidden-case accuracy	90.50%	Setup & source Exact named hosted API observation; Hidden-case accuracy. Named source route baseten/mammoth-1; ModelTested runner v1; tested_on=openrouter. Backend weight revision and API inference precision unpublished. Publisher generated 2026-11-01; board added this route 2026-10-29. Execution dates, token usage and hashes remain in its per-run receipts. No private prompt/response archive imported. 56 authored situations, including 19 hard cases, each with two typed questions; 14 public and 42 hidden cases. Source aggregate averages per-case decision accuracy. Confidence score is 100×(1−mean multiclass Brier/2), not ECE or a calibration probability. Small single-run benchmark; current HF access/size does not inherit this evaluator’s stale closed/API-only metadata. No tariff inferred from rounded cost 0.0. Rows added 1 November 2026 use revision 668a52ad3b932ca910fd985f96dcdf0a85d2dae1, where the earlier 11 systems are unchanged. Rows added 2 November 2026 use revision 5c6c19bb3b7332ccd28c8c13bd32cdc69cedc8f0. Checked / reported: 2026-11-01 · Samples: 56 Snapshot: 2026-10-18 · Revision: 081e8f37130a4a9077b8cffe62827d4c3b9e3b23; 668a52ad3b932ca910fd985f96dcdf0a85d2dae1 for Drex Max, Jev 1.14 and d1.1; 5c6c19bb3b7332ccd28c8c13bd32cdc69cedc8f0 for Mammoth, Elephant, Joey, Span-02 and both Solar Decide routes · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
ModelTested 56-case · Brier-based confidence score	95.40	Setup & source Exact named hosted API observation; Brier-based confidence score. Named source route baseten/mammoth-1; ModelTested runner v1; tested_on=openrouter. Backend weight revision and API inference precision unpublished. Publisher generated 2026-11-01; board added this route 2026-10-29. Execution dates, token usage and hashes remain in its per-run receipts. No private prompt/response archive imported. 56 authored situations, including 19 hard cases, each with two typed questions; 14 public and 42 hidden cases. Source aggregate averages per-case decision accuracy. Confidence score is 100×(1−mean multiclass Brier/2), not ECE or a calibration probability. Small single-run benchmark; current HF access/size does not inherit this evaluator’s stale closed/API-only metadata. No tariff inferred from rounded cost 0.0. Rows added 1 November 2026 use revision 668a52ad3b932ca910fd985f96dcdf0a85d2dae1, where the earlier 11 systems are unchanged. Rows added 2 November 2026 use revision 5c6c19bb3b7332ccd28c8c13bd32cdc69cedc8f0. Checked / reported: 2026-11-01 · Samples: 56 Snapshot: 2026-10-18 · Revision: 081e8f37130a4a9077b8cffe62827d4c3b9e3b23; 668a52ad3b932ca910fd985f96dcdf0a85d2dae1 for Drex Max, Jev 1.14 and d1.1; 5c6c19bb3b7332ccd28c8c13bd32cdc69cedc8f0 for Mammoth, Elephant, Joey, Span-02 and both Solar Decide routes · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source

Independent · Typed Decisions · card owner

mammoth-1 · Independent · Typed Decisions · card owner
Evaluation	Result	Evidence
Typed Decisions · Scored by the card owner via OpenRouter · Accuracy	76.80%	Setup & source Accuracy for a general zero-shot row scored by the card owner. OpenRouter route baseten/mammoth-1:free, source label general, zero-shot; scored by the card owner. Weight revision and hardware not stated. Published on a 0–1 scale. Rows the card owner lists as scored by themselves through OpenRouter on the free route, added 22 October 2026. The card does not state the served weight revision. Separate from the SaylorTwift and creator-reported families. Checked / reported: 2026-11-02 · Samples: 2000 Snapshot: 2026-10-22T11:19:26.000Z · Revision: fa01e869d676b3098f6d1546f3c6594fa1230ea2 · Apache-2.0 dataset/card; third-party published observations Result source · Benchmark source
Typed Decisions · Scored by the card owner via OpenRouter · KL from gold	0.122 KL	Setup & source KL from gold for a general zero-shot row scored by the card owner. OpenRouter route baseten/mammoth-1:free, source label general, zero-shot; scored by the card owner. Weight revision and hardware not stated. Rows the card owner lists as scored by themselves through OpenRouter on the free route, added 22 October 2026. The card does not state the served weight revision. Separate from the SaylorTwift and creator-reported families. Checked / reported: 2026-11-02 · Samples: 2000 Snapshot: 2026-10-22T11:19:26.000Z · Revision: fa01e869d676b3098f6d1546f3c6594fa1230ea2 · Apache-2.0 dataset/card; third-party published observations Result source · Benchmark source
Typed Decisions · Scored by the card owner via OpenRouter · Brier	0.063 Brier	Setup & source Brier for a general zero-shot row scored by the card owner. OpenRouter route baseten/mammoth-1:free, source label general, zero-shot; scored by the card owner. Weight revision and hardware not stated. Rows the card owner lists as scored by themselves through OpenRouter on the free route, added 22 October 2026. The card does not state the served weight revision. Separate from the SaylorTwift and creator-reported families. Checked / reported: 2026-11-02 · Samples: 2000 Snapshot: 2026-10-22T11:19:26.000Z · Revision: fa01e869d676b3098f6d1546f3c6594fa1230ea2 · Apache-2.0 dataset/card; third-party published observations Result source · Benchmark source
Typed Decisions · Scored by the card owner via OpenRouter · ECE	15.30 pp	Setup & source ECE for a general zero-shot row scored by the card owner. OpenRouter route baseten/mammoth-1:free, source label general, zero-shot; scored by the card owner. Weight revision and hardware not stated. Published on a 0–1 scale. Rows the card owner lists as scored by themselves through OpenRouter on the free route, added 22 October 2026. The card does not state the served weight revision. Separate from the SaylorTwift and creator-reported families. Checked / reported: 2026-11-02 · Samples: 2000 Snapshot: 2026-10-22T11:19:26.000Z · Revision: fa01e869d676b3098f6d1546f3c6594fa1230ea2 · Apache-2.0 dataset/card; third-party published observations Result source · Benchmark source
Typed Decisions · Scored by the card owner via OpenRouter · Median latency	415 ms	Setup & source Median latency through the OpenRouter free route, network included. OpenRouter route baseten/mammoth-1:free, source label general, zero-shot; scored by the card owner. Weight revision and hardware not stated. Free tier through OpenRouter; not a controlled timing. Rows the card owner lists as scored by themselves through OpenRouter on the free route, added 22 October 2026. The card does not state the served weight revision. Separate from the SaylorTwift and creator-reported families. Checked / reported: 2026-11-02 · Samples: 2000 Snapshot: 2026-10-22T11:19:26.000Z · Revision: fa01e869d676b3098f6d1546f3c6594fa1230ea2 · Apache-2.0 dataset/card; third-party published observations Result source · Benchmark source

Evaluator-published · Benchmark Heaven

mammoth-1 · Evaluator-published · Benchmark Heaven
Evaluation	Result	Evidence
Benchmark Heaven JevBench v1.8 · Capability Score	85.233	Setup & source Capability Score on the rendered board. baselabs mammoth-1 (27B) · Open weights · LLM decoder. Board key mammoth-1; method tag v1.7. Measured on the Second regular draw (v1.7.0), 2026-10-22; put on the v1.6.1 scale by the draw's anchor offset (see How it works). Source link https://huggingface.co/baselabs/mammoth-1. Rendered board row. Benchmark Heaven's rendered board, read 2 November 2026: 283 ranked systems · 1,500 decisions each (600 for 19 equated API re-runs) · 131 newer ranked rows tagged with their draw · 14 older rows dated separately. Newer rows are measured on dated draws and put on the v1.6.1 scale by the draw's anchor offset, so a row's draw is part of its identity. Distinct from Leanmcp JevBench, the community Decision Index and the stored Benchmark Heaven v1.6.1 feed. Cost is the source's model on a GPU-time basis for self-hosted rows, not a provider tariff. Self-hosted latency is timed on the GPU over loopback and API latency end to end with the network; the source says they are not directly comparable, so they stay in separate datasets. No Examenos recomputation. Checked / reported: 2026-11-02 · Evaluated: 2026-10-22 · Samples: Not published Snapshot: 2026-10-31 · Revision: Board v1.7.48, method v1.8, updated 2026-10-31; page sha256 df9093384c9cdf5854e065f71dd3abd83a127793f22f4ca70268e9981e0a0b17 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
Benchmark Heaven JevBench v1.8 · Intelligence	76.2	Setup & source Intelligence on the rendered board. baselabs mammoth-1 (27B) · Open weights · LLM decoder. Board key mammoth-1; method tag v1.7. Measured on the Second regular draw (v1.7.0), 2026-10-22; put on the v1.6.1 scale by the draw's anchor offset (see How it works). Source link https://huggingface.co/baselabs/mammoth-1. Rendered board row. Benchmark Heaven's rendered board, read 2 November 2026: 283 ranked systems · 1,500 decisions each (600 for 19 equated API re-runs) · 131 newer ranked rows tagged with their draw · 14 older rows dated separately. Newer rows are measured on dated draws and put on the v1.6.1 scale by the draw's anchor offset, so a row's draw is part of its identity. Distinct from Leanmcp JevBench, the community Decision Index and the stored Benchmark Heaven v1.6.1 feed. Cost is the source's model on a GPU-time basis for self-hosted rows, not a provider tariff. Self-hosted latency is timed on the GPU over loopback and API latency end to end with the network; the source says they are not directly comparable, so they stay in separate datasets. No Examenos recomputation. Checked / reported: 2026-11-02 · Evaluated: 2026-10-22 · Samples: Not published Snapshot: 2026-10-31 · Revision: Board v1.7.48, method v1.8, updated 2026-10-31; page sha256 df9093384c9cdf5854e065f71dd3abd83a127793f22f4ca70268e9981e0a0b17 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
Benchmark Heaven JevBench v1.8 · Calibration	94.3	Setup & source Calibration on the rendered board. baselabs mammoth-1 (27B) · Open weights · LLM decoder. Board key mammoth-1; method tag v1.7. Measured on the Second regular draw (v1.7.0), 2026-10-22; put on the v1.6.1 scale by the draw's anchor offset (see How it works). Source link https://huggingface.co/baselabs/mammoth-1. Rendered board row. Benchmark Heaven's rendered board, read 2 November 2026: 283 ranked systems · 1,500 decisions each (600 for 19 equated API re-runs) · 131 newer ranked rows tagged with their draw · 14 older rows dated separately. Newer rows are measured on dated draws and put on the v1.6.1 scale by the draw's anchor offset, so a row's draw is part of its identity. Distinct from Leanmcp JevBench, the community Decision Index and the stored Benchmark Heaven v1.6.1 feed. Cost is the source's model on a GPU-time basis for self-hosted rows, not a provider tariff. Self-hosted latency is timed on the GPU over loopback and API latency end to end with the network; the source says they are not directly comparable, so they stay in separate datasets. No Examenos recomputation. Checked / reported: 2026-11-02 · Evaluated: 2026-10-22 · Samples: Not published Snapshot: 2026-10-31 · Revision: Board v1.7.48, method v1.8, updated 2026-10-31; page sha256 df9093384c9cdf5854e065f71dd3abd83a127793f22f4ca70268e9981e0a0b17 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
Benchmark Heaven JevBench v1.8 · Cost Score	57.2	Setup & source Cost Score on the rendered board. baselabs mammoth-1 (27B) · Open weights · LLM decoder. Board key mammoth-1; method tag v1.7. Measured on the Second regular draw (v1.7.0), 2026-10-22; put on the v1.6.1 scale by the draw's anchor offset (see How it works). Source link https://huggingface.co/baselabs/mammoth-1. Rendered board row. Benchmark Heaven's rendered board, read 2 November 2026: 283 ranked systems · 1,500 decisions each (600 for 19 equated API re-runs) · 131 newer ranked rows tagged with their draw · 14 older rows dated separately. Newer rows are measured on dated draws and put on the v1.6.1 scale by the draw's anchor offset, so a row's draw is part of its identity. Distinct from Leanmcp JevBench, the community Decision Index and the stored Benchmark Heaven v1.6.1 feed. Cost is the source's model on a GPU-time basis for self-hosted rows, not a provider tariff. Self-hosted latency is timed on the GPU over loopback and API latency end to end with the network; the source says they are not directly comparable, so they stay in separate datasets. No Examenos recomputation. Checked / reported: 2026-11-02 · Evaluated: 2026-10-22 · Samples: Not published Snapshot: 2026-10-31 · Revision: Board v1.7.48, method v1.8, updated 2026-10-31; page sha256 df9093384c9cdf5854e065f71dd3abd83a127793f22f4ca70268e9981e0a0b17 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
Benchmark Heaven JevBench v1.8 · Modelled cost per 1,000 decisions	0.02678 USD/1k decisions	Setup & source Modelled cost per 1,000 decisions on the rendered board. baselabs mammoth-1 (27B) · Open weights · LLM decoder. Board key mammoth-1; method tag v1.7. Measured on the Second regular draw (v1.7.0), 2026-10-22; put on the v1.6.1 scale by the draw's anchor offset (see How it works). Source link https://huggingface.co/baselabs/mammoth-1. Rendered board row. Benchmark Heaven's rendered board, read 2 November 2026: 283 ranked systems · 1,500 decisions each (600 for 19 equated API re-runs) · 131 newer ranked rows tagged with their draw · 14 older rows dated separately. Newer rows are measured on dated draws and put on the v1.6.1 scale by the draw's anchor offset, so a row's draw is part of its identity. Distinct from Leanmcp JevBench, the community Decision Index and the stored Benchmark Heaven v1.6.1 feed. Cost is the source's model on a GPU-time basis for self-hosted rows, not a provider tariff. Self-hosted latency is timed on the GPU over loopback and API latency end to end with the network; the source says they are not directly comparable, so they stay in separate datasets. No Examenos recomputation. Checked / reported: 2026-11-02 · Evaluated: 2026-10-22 · Samples: Not published Snapshot: 2026-10-31 · Revision: Board v1.7.48, method v1.8, updated 2026-10-31; page sha256 df9093384c9cdf5854e065f71dd3abd83a127793f22f4ca70268e9981e0a0b17 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
Benchmark Heaven JevBench v1.8 · Self-hosted p50 on the GPU, loopback	0.188 s	Setup & source Self-hosted p50 on the GPU, loopback on the rendered board. baselabs mammoth-1 (27B) · Open weights · LLM decoder. Board key mammoth-1; method tag v1.7. Measured on the Second regular draw (v1.7.0), 2026-10-22; put on the v1.6.1 scale by the draw's anchor offset (see How it works). Source link https://huggingface.co/baselabs/mammoth-1. Rendered board row. Benchmark Heaven's rendered board, read 2 November 2026: 283 ranked systems · 1,500 decisions each (600 for 19 equated API re-runs) · 131 newer ranked rows tagged with their draw · 14 older rows dated separately. Newer rows are measured on dated draws and put on the v1.6.1 scale by the draw's anchor offset, so a row's draw is part of its identity. Distinct from Leanmcp JevBench, the community Decision Index and the stored Benchmark Heaven v1.6.1 feed. Cost is the source's model on a GPU-time basis for self-hosted rows, not a provider tariff. Self-hosted latency is timed on the GPU over loopback and API latency end to end with the network; the source says they are not directly comparable, so they stay in separate datasets. No Examenos recomputation. Checked / reported: 2026-11-02 · Evaluated: 2026-10-22 · Samples: Not published Snapshot: 2026-10-31 · Revision: Board v1.7.48, method v1.8, updated 2026-10-31; page sha256 df9093384c9cdf5854e065f71dd3abd83a127793f22f4ca70268e9981e0a0b17 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
Benchmark Heaven JevBench v1.8 · Self-hosted estimate including network (2x + 150 ms)	0.53 s	Setup & source Self-hosted estimate including network (2x + 150 ms) on the rendered board. baselabs mammoth-1 (27B) · Open weights · LLM decoder. Board key mammoth-1; method tag v1.7. Measured on the Second regular draw (v1.7.0), 2026-10-22; put on the v1.6.1 scale by the draw's anchor offset (see How it works). Source link https://huggingface.co/baselabs/mammoth-1. Rendered board row. Benchmark Heaven's rendered board, read 2 November 2026: 283 ranked systems · 1,500 decisions each (600 for 19 equated API re-runs) · 131 newer ranked rows tagged with their draw · 14 older rows dated separately. Newer rows are measured on dated draws and put on the v1.6.1 scale by the draw's anchor offset, so a row's draw is part of its identity. Distinct from Leanmcp JevBench, the community Decision Index and the stored Benchmark Heaven v1.6.1 feed. Cost is the source's model on a GPU-time basis for self-hosted rows, not a provider tariff. Self-hosted latency is timed on the GPU over loopback and API latency end to end with the network; the source says they are not directly comparable, so they stay in separate datasets. No Examenos recomputation. Checked / reported: 2026-11-02 · Evaluated: 2026-10-22 · Samples: Not published Snapshot: 2026-10-31 · Revision: Board v1.7.48, method v1.8, updated 2026-10-31; page sha256 df9093384c9cdf5854e065f71dd3abd83a127793f22f4ca70268e9981e0a0b17 · Published summary observations; upstream task/result terms retained Result source · Benchmark source

mammoth-1 · Evaluator-published · Community Decision Index
Evaluation	Result	Evidence
HF text 0.4.1 · Full score	68.44	Setup & source HF text 0.4.1 · Full score Exact source engine mammoth-1; {"kind": "full fine-tune", "technique": "autoregressive", "technique_name": "autoregressive", "served_params": 27356728560, "params_estimated": false, "params_from_name": false, "trained_bytes": null, "model_url": "https://huggingface.co/baselabs/mammoth-1", "code_url": null, "base_model": "Qwen/Qwen3.5-27B", "base_url": "https://huggingface.co/Qwen/Qwen3.5-27B", "weights_repo": "baselabs/mammoth-1", "served_checkpoint": "9d1962f193e48a68cf0f3598dc7fb4462fd23ad9", "org": {"id": "baselabs", "name": "baselabs", "logo": "data/orgs/baselabs.png"}, "pr": 260, "new": true, "show_notes": true}. New in the 2026-10-18 PR batch (PR #244); updated to revision 9d1962f in PR #260. Public 0.3 run from PR #260 adopted (kit rescore 75.34, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving at revision 8f9f8619, whose weights are identical to 9d1962f; decision agreement with the submitted rows 99.85% on 831 rows (bar 98.5%). Weights are public on the Hugging Face Hub under a proprietary license. Training data not disclosed. One private new-domain test is not counted for this model: our checks found signs its training data overlapped it. Its new-domain score uses the remaining tests, put on the same scale with the same stock models. Decision Index 0.4.1 (release-v4), source generated 2026-10-29T14:04:16Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Source notes: New in the 2026-10-18 PR batch (PR #244); updated to revision 9d1962f in PR #260. Public 0.3 run from PR #260 adopted (kit rescore 75.34, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving at revision 8f9f8619, whose weights are identical to 9d1962f; decision agreement with the submitted rows 99.85% on 831 rows (bar 98.5%). Weights are public on the Hugging Face Hub under a proprietary license. Training data not disclosed. One private new-domain test is not counted for this model: our checks found signs its training data overlapped it. Its new-domain score uses the remaining tests, put on the same scale with the same stock models. Author public-run adoption/execution status and private lab setup are disclosed in configuration; evaluator-published does not mean every public run was independently executed. The board served 9d1962f193; the registry pins the newest repository revision, and the commit titles after it concern the card. Community Decision Index 0.4.1: expanded/refreshed harder private new-domain tasks and equating; Full remains 60% private same-skill +40% private new domains, zero public contribution. This population revision is not a model upgrade/regression. Public/native/calibration data remain on their unchanged 0.4 definitions. Source publication/generation is not a model run date. Public submitted runs plus evaluator private measurements are Evaluator-published, not wholly independently executed. Checked / reported: 2026-11-02 · Samples: Not published Snapshot: 2026-10-22T05:40:06Z · Revision: d9962113a1acc95c2d7522c2babfb989afbeb034; c130faf65f82e81d3bdefb01e758c712dc5c5199 for the 2 November 2026 additions · Published observations; result reuse terms not specified Result source · Benchmark source
HF text 0.4 · Public index	75.34	Setup & source HF text 0.4 · Public index Exact source engine mammoth-1; {"kind": "full fine-tune", "technique": "autoregressive", "technique_name": "autoregressive", "served_params": 27356728560, "params_estimated": false, "params_from_name": false, "trained_bytes": null, "model_url": "https://huggingface.co/baselabs/mammoth-1", "code_url": null, "base_model": "Qwen/Qwen3.5-27B", "base_url": "https://huggingface.co/Qwen/Qwen3.5-27B", "weights_repo": "baselabs/mammoth-1", "served_checkpoint": "9d1962f193e48a68cf0f3598dc7fb4462fd23ad9", "org": {"id": "baselabs", "name": "baselabs", "logo": "data/orgs/baselabs.png"}, "pr": 260, "new": true, "show_notes": true}. New in the 2026-10-18 PR batch (PR #244); updated to revision 9d1962f in PR #260. Public 0.3 run from PR #260 adopted (kit rescore 75.34, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving at revision 8f9f8619, whose weights are identical to 9d1962f; decision agreement with the submitted rows 99.85% on 831 rows (bar 98.5%). Weights are public on the Hugging Face Hub under a proprietary license. Training data not disclosed. One private new-domain test is not counted for this model: our checks found signs its training data overlapped it. Its new-domain score uses the remaining tests, put on the same scale with the same stock models. Decision Index 0.4.1 (release-v4), source generated 2026-10-29T14:04:16Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Source notes: New in the 2026-10-18 PR batch (PR #244); updated to revision 9d1962f in PR #260. Public 0.3 run from PR #260 adopted (kit rescore 75.34, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving at revision 8f9f8619, whose weights are identical to 9d1962f; decision agreement with the submitted rows 99.85% on 831 rows (bar 98.5%). Weights are public on the Hugging Face Hub under a proprietary license. Training data not disclosed. One private new-domain test is not counted for this model: our checks found signs its training data overlapped it. Its new-domain score uses the remaining tests, put on the same scale with the same stock models. Author public-run adoption/execution status and private lab setup are disclosed in configuration; evaluator-published does not mean every public run was independently executed. The board served 9d1962f193; the registry pins the newest repository revision, and the commit titles after it concern the card. Decision Index 0.4 (release-v4), source generated 2026-10-20T14:13:56Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Checked / reported: 2026-11-02 · Samples: Not published Snapshot: 2026-10-20T14:13:56Z · Revision: 133313cf8963f2b6bed960abb0152fd14cbb6731; c130faf65f82e81d3bdefb01e758c712dc5c5199 for the 2 November 2026 additions · Published observations; result reuse terms not specified Result source · Benchmark source
HF text 0.4 · Private same-skill component	66.26	Setup & source HF text 0.4 · Private same-skill component Exact source engine mammoth-1; {"kind": "full fine-tune", "technique": "autoregressive", "technique_name": "autoregressive", "served_params": 27356728560, "params_estimated": false, "params_from_name": false, "trained_bytes": null, "model_url": "https://huggingface.co/baselabs/mammoth-1", "code_url": null, "base_model": "Qwen/Qwen3.5-27B", "base_url": "https://huggingface.co/Qwen/Qwen3.5-27B", "weights_repo": "baselabs/mammoth-1", "served_checkpoint": "9d1962f193e48a68cf0f3598dc7fb4462fd23ad9", "org": {"id": "baselabs", "name": "baselabs", "logo": "data/orgs/baselabs.png"}, "pr": 260, "new": true, "show_notes": true}. New in the 2026-10-18 PR batch (PR #244); updated to revision 9d1962f in PR #260. Public 0.3 run from PR #260 adopted (kit rescore 75.34, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving at revision 8f9f8619, whose weights are identical to 9d1962f; decision agreement with the submitted rows 99.85% on 831 rows (bar 98.5%). Weights are public on the Hugging Face Hub under a proprietary license. Training data not disclosed. One private new-domain test is not counted for this model: our checks found signs its training data overlapped it. Its new-domain score uses the remaining tests, put on the same scale with the same stock models. Decision Index 0.4.1 (release-v4), source generated 2026-10-29T14:04:16Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Source notes: New in the 2026-10-18 PR batch (PR #244); updated to revision 9d1962f in PR #260. Public 0.3 run from PR #260 adopted (kit rescore 75.34, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving at revision 8f9f8619, whose weights are identical to 9d1962f; decision agreement with the submitted rows 99.85% on 831 rows (bar 98.5%). Weights are public on the Hugging Face Hub under a proprietary license. Training data not disclosed. One private new-domain test is not counted for this model: our checks found signs its training data overlapped it. Its new-domain score uses the remaining tests, put on the same scale with the same stock models. Author public-run adoption/execution status and private lab setup are disclosed in configuration; evaluator-published does not mean every public run was independently executed. The board served 9d1962f193; the registry pins the newest repository revision, and the commit titles after it concern the card. Decision Index 0.4 (release-v4), source generated 2026-10-20T14:13:56Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Checked / reported: 2026-11-02 · Samples: Not published Snapshot: 2026-10-20T14:13:56Z · Revision: 133313cf8963f2b6bed960abb0152fd14cbb6731; c130faf65f82e81d3bdefb01e758c712dc5c5199 for the 2 November 2026 additions · Published observations; result reuse terms not specified Result source · Benchmark source
HF text 0.4.1 · Private new-domain component	56.27	Setup & source HF text 0.4.1 · Private new-domain component Exact source engine mammoth-1; {"kind": "full fine-tune", "technique": "autoregressive", "technique_name": "autoregressive", "served_params": 27356728560, "params_estimated": false, "params_from_name": false, "trained_bytes": null, "model_url": "https://huggingface.co/baselabs/mammoth-1", "code_url": null, "base_model": "Qwen/Qwen3.5-27B", "base_url": "https://huggingface.co/Qwen/Qwen3.5-27B", "weights_repo": "baselabs/mammoth-1", "served_checkpoint": "9d1962f193e48a68cf0f3598dc7fb4462fd23ad9", "org": {"id": "baselabs", "name": "baselabs", "logo": "data/orgs/baselabs.png"}, "pr": 260, "new": true, "show_notes": true}. New in the 2026-10-18 PR batch (PR #244); updated to revision 9d1962f in PR #260. Public 0.3 run from PR #260 adopted (kit rescore 75.34, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving at revision 8f9f8619, whose weights are identical to 9d1962f; decision agreement with the submitted rows 99.85% on 831 rows (bar 98.5%). Weights are public on the Hugging Face Hub under a proprietary license. Training data not disclosed. One private new-domain test is not counted for this model: our checks found signs its training data overlapped it. Its new-domain score uses the remaining tests, put on the same scale with the same stock models. Decision Index 0.4.1 (release-v4), source generated 2026-10-29T14:04:16Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Source notes: New in the 2026-10-18 PR batch (PR #244); updated to revision 9d1962f in PR #260. Public 0.3 run from PR #260 adopted (kit rescore 75.34, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving at revision 8f9f8619, whose weights are identical to 9d1962f; decision agreement with the submitted rows 99.85% on 831 rows (bar 98.5%). Weights are public on the Hugging Face Hub under a proprietary license. Training data not disclosed. One private new-domain test is not counted for this model: our checks found signs its training data overlapped it. Its new-domain score uses the remaining tests, put on the same scale with the same stock models. Author public-run adoption/execution status and private lab setup are disclosed in configuration; evaluator-published does not mean every public run was independently executed. The board served 9d1962f193; the registry pins the newest repository revision, and the commit titles after it concern the card. Community Decision Index 0.4.1: expanded/refreshed harder private new-domain tasks and equating; Full remains 60% private same-skill +40% private new domains, zero public contribution. This population revision is not a model upgrade/regression. Public/native/calibration data remain on their unchanged 0.4 definitions. Source publication/generation is not a model run date. Public submitted runs plus evaluator private measurements are Evaluator-published, not wholly independently executed. Checked / reported: 2026-11-02 · Samples: Not published Snapshot: 2026-10-22T05:40:06Z · Revision: d9962113a1acc95c2d7522c2babfb989afbeb034; c130faf65f82e81d3bdefb01e758c712dc5c5199 for the 2 November 2026 additions · Published observations; result reuse terms not specified Result source · Benchmark source
HF text 0.4 · Raw public index	81.19	Setup & source HF text 0.4 · Raw public index Exact source engine mammoth-1; {"kind": "full fine-tune", "technique": "autoregressive", "technique_name": "autoregressive", "served_params": 27356728560, "params_estimated": false, "params_from_name": false, "trained_bytes": null, "model_url": "https://huggingface.co/baselabs/mammoth-1", "code_url": null, "base_model": "Qwen/Qwen3.5-27B", "base_url": "https://huggingface.co/Qwen/Qwen3.5-27B", "weights_repo": "baselabs/mammoth-1", "served_checkpoint": "9d1962f193e48a68cf0f3598dc7fb4462fd23ad9", "org": {"id": "baselabs", "name": "baselabs", "logo": "data/orgs/baselabs.png"}, "pr": 260, "new": true, "show_notes": true}. New in the 2026-10-18 PR batch (PR #244); updated to revision 9d1962f in PR #260. Public 0.3 run from PR #260 adopted (kit rescore 75.34, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving at revision 8f9f8619, whose weights are identical to 9d1962f; decision agreement with the submitted rows 99.85% on 831 rows (bar 98.5%). Weights are public on the Hugging Face Hub under a proprietary license. Training data not disclosed. One private new-domain test is not counted for this model: our checks found signs its training data overlapped it. Its new-domain score uses the remaining tests, put on the same scale with the same stock models. Decision Index 0.4.1 (release-v4), source generated 2026-10-29T14:04:16Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Source notes: New in the 2026-10-18 PR batch (PR #244); updated to revision 9d1962f in PR #260. Public 0.3 run from PR #260 adopted (kit rescore 75.34, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving at revision 8f9f8619, whose weights are identical to 9d1962f; decision agreement with the submitted rows 99.85% on 831 rows (bar 98.5%). Weights are public on the Hugging Face Hub under a proprietary license. Training data not disclosed. One private new-domain test is not counted for this model: our checks found signs its training data overlapped it. Its new-domain score uses the remaining tests, put on the same scale with the same stock models. Author public-run adoption/execution status and private lab setup are disclosed in configuration; evaluator-published does not mean every public run was independently executed. The board served 9d1962f193; the registry pins the newest repository revision, and the commit titles after it concern the card. Decision Index 0.4 (release-v4), source generated 2026-10-20T14:13:56Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Checked / reported: 2026-11-02 · Samples: Not published Snapshot: 2026-10-20T14:13:56Z · Revision: 133313cf8963f2b6bed960abb0152fd14cbb6731; c130faf65f82e81d3bdefb01e758c712dc5c5199 for the 2 November 2026 additions · Published observations; result reuse terms not specified Result source · Benchmark source
HF text 0.4 · Public breadth index	75.00	Setup & source HF text 0.4 · Public breadth index Exact source engine mammoth-1; {"kind": "full fine-tune", "technique": "autoregressive", "technique_name": "autoregressive", "served_params": 27356728560, "params_estimated": false, "params_from_name": false, "trained_bytes": null, "model_url": "https://huggingface.co/baselabs/mammoth-1", "code_url": null, "base_model": "Qwen/Qwen3.5-27B", "base_url": "https://huggingface.co/Qwen/Qwen3.5-27B", "weights_repo": "baselabs/mammoth-1", "served_checkpoint": "9d1962f193e48a68cf0f3598dc7fb4462fd23ad9", "org": {"id": "baselabs", "name": "baselabs", "logo": "data/orgs/baselabs.png"}, "pr": 260, "new": true, "show_notes": true}. New in the 2026-10-18 PR batch (PR #244); updated to revision 9d1962f in PR #260. Public 0.3 run from PR #260 adopted (kit rescore 75.34, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving at revision 8f9f8619, whose weights are identical to 9d1962f; decision agreement with the submitted rows 99.85% on 831 rows (bar 98.5%). Weights are public on the Hugging Face Hub under a proprietary license. Training data not disclosed. One private new-domain test is not counted for this model: our checks found signs its training data overlapped it. Its new-domain score uses the remaining tests, put on the same scale with the same stock models. Decision Index 0.4.1 (release-v4), source generated 2026-10-29T14:04:16Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Source notes: New in the 2026-10-18 PR batch (PR #244); updated to revision 9d1962f in PR #260. Public 0.3 run from PR #260 adopted (kit rescore 75.34, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving at revision 8f9f8619, whose weights are identical to 9d1962f; decision agreement with the submitted rows 99.85% on 831 rows (bar 98.5%). Weights are public on the Hugging Face Hub under a proprietary license. Training data not disclosed. One private new-domain test is not counted for this model: our checks found signs its training data overlapped it. Its new-domain score uses the remaining tests, put on the same scale with the same stock models. Author public-run adoption/execution status and private lab setup are disclosed in configuration; evaluator-published does not mean every public run was independently executed. The board served 9d1962f193; the registry pins the newest repository revision, and the commit titles after it concern the card. Decision Index 0.4 (release-v4), source generated 2026-10-20T14:13:56Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Checked / reported: 2026-11-02 · Samples: Not published Snapshot: 2026-10-20T14:13:56Z · Revision: 133313cf8963f2b6bed960abb0152fd14cbb6731; c130faf65f82e81d3bdefb01e758c712dc5c5199 for the 2 November 2026 additions · Published observations; result reuse terms not specified Result source · Benchmark source
HF text 0.4 · Equated private same-skill	65.98	Setup & source HF text 0.4 · Equated private same-skill Exact source engine mammoth-1; {"kind": "full fine-tune", "technique": "autoregressive", "technique_name": "autoregressive", "served_params": 27356728560, "params_estimated": false, "params_from_name": false, "trained_bytes": null, "model_url": "https://huggingface.co/baselabs/mammoth-1", "code_url": null, "base_model": "Qwen/Qwen3.5-27B", "base_url": "https://huggingface.co/Qwen/Qwen3.5-27B", "weights_repo": "baselabs/mammoth-1", "served_checkpoint": "9d1962f193e48a68cf0f3598dc7fb4462fd23ad9", "org": {"id": "baselabs", "name": "baselabs", "logo": "data/orgs/baselabs.png"}, "pr": 260, "new": true, "show_notes": true}. New in the 2026-10-18 PR batch (PR #244); updated to revision 9d1962f in PR #260. Public 0.3 run from PR #260 adopted (kit rescore 75.34, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving at revision 8f9f8619, whose weights are identical to 9d1962f; decision agreement with the submitted rows 99.85% on 831 rows (bar 98.5%). Weights are public on the Hugging Face Hub under a proprietary license. Training data not disclosed. One private new-domain test is not counted for this model: our checks found signs its training data overlapped it. Its new-domain score uses the remaining tests, put on the same scale with the same stock models. Decision Index 0.4.1 (release-v4), source generated 2026-10-29T14:04:16Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Source notes: New in the 2026-10-18 PR batch (PR #244); updated to revision 9d1962f in PR #260. Public 0.3 run from PR #260 adopted (kit rescore 75.34, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving at revision 8f9f8619, whose weights are identical to 9d1962f; decision agreement with the submitted rows 99.85% on 831 rows (bar 98.5%). Weights are public on the Hugging Face Hub under a proprietary license. Training data not disclosed. One private new-domain test is not counted for this model: our checks found signs its training data overlapped it. Its new-domain score uses the remaining tests, put on the same scale with the same stock models. Author public-run adoption/execution status and private lab setup are disclosed in configuration; evaluator-published does not mean every public run was independently executed. The board served 9d1962f193; the registry pins the newest repository revision, and the commit titles after it concern the card. Decision Index 0.4 (release-v4), source generated 2026-10-20T14:13:56Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Checked / reported: 2026-11-02 · Samples: Not published Snapshot: 2026-10-20T14:13:56Z · Revision: 133313cf8963f2b6bed960abb0152fd14cbb6731; c130faf65f82e81d3bdefb01e758c712dc5c5199 for the 2 November 2026 additions · Published observations; result reuse terms not specified Result source · Benchmark source
HF text 0.4.1 · Equated private new domains	72.14	Setup & source HF text 0.4.1 · Equated private new domains Exact source engine mammoth-1; {"kind": "full fine-tune", "technique": "autoregressive", "technique_name": "autoregressive", "served_params": 27356728560, "params_estimated": false, "params_from_name": false, "trained_bytes": null, "model_url": "https://huggingface.co/baselabs/mammoth-1", "code_url": null, "base_model": "Qwen/Qwen3.5-27B", "base_url": "https://huggingface.co/Qwen/Qwen3.5-27B", "weights_repo": "baselabs/mammoth-1", "served_checkpoint": "9d1962f193e48a68cf0f3598dc7fb4462fd23ad9", "org": {"id": "baselabs", "name": "baselabs", "logo": "data/orgs/baselabs.png"}, "pr": 260, "new": true, "show_notes": true}. New in the 2026-10-18 PR batch (PR #244); updated to revision 9d1962f in PR #260. Public 0.3 run from PR #260 adopted (kit rescore 75.34, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving at revision 8f9f8619, whose weights are identical to 9d1962f; decision agreement with the submitted rows 99.85% on 831 rows (bar 98.5%). Weights are public on the Hugging Face Hub under a proprietary license. Training data not disclosed. One private new-domain test is not counted for this model: our checks found signs its training data overlapped it. Its new-domain score uses the remaining tests, put on the same scale with the same stock models. Decision Index 0.4.1 (release-v4), source generated 2026-10-29T14:04:16Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Source notes: New in the 2026-10-18 PR batch (PR #244); updated to revision 9d1962f in PR #260. Public 0.3 run from PR #260 adopted (kit rescore 75.34, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving at revision 8f9f8619, whose weights are identical to 9d1962f; decision agreement with the submitted rows 99.85% on 831 rows (bar 98.5%). Weights are public on the Hugging Face Hub under a proprietary license. Training data not disclosed. One private new-domain test is not counted for this model: our checks found signs its training data overlapped it. Its new-domain score uses the remaining tests, put on the same scale with the same stock models. Author public-run adoption/execution status and private lab setup are disclosed in configuration; evaluator-published does not mean every public run was independently executed. The board served 9d1962f193; the registry pins the newest repository revision, and the commit titles after it concern the card. Community Decision Index 0.4.1: expanded/refreshed harder private new-domain tasks and equating; Full remains 60% private same-skill +40% private new domains, zero public contribution. This population revision is not a model upgrade/regression. Public/native/calibration data remain on their unchanged 0.4 definitions. Source publication/generation is not a model run date. Public submitted runs plus evaluator private measurements are Evaluator-published, not wholly independently executed. Checked / reported: 2026-11-02 · Samples: Not published Snapshot: 2026-10-22T05:40:06Z · Revision: d9962113a1acc95c2d7522c2babfb989afbeb034; c130faf65f82e81d3bdefb01e758c712dc5c5199 for the 2 November 2026 additions · Published observations; result reuse terms not specified Result source · Benchmark source
HF text 0.4 · Full contribution · public	0.00	Setup & source HF text 0.4 · Full contribution · public Exact source engine mammoth-1; {"kind": "full fine-tune", "technique": "autoregressive", "technique_name": "autoregressive", "served_params": 27356728560, "params_estimated": false, "params_from_name": false, "trained_bytes": null, "model_url": "https://huggingface.co/baselabs/mammoth-1", "code_url": null, "base_model": "Qwen/Qwen3.5-27B", "base_url": "https://huggingface.co/Qwen/Qwen3.5-27B", "weights_repo": "baselabs/mammoth-1", "served_checkpoint": "9d1962f193e48a68cf0f3598dc7fb4462fd23ad9", "org": {"id": "baselabs", "name": "baselabs", "logo": "data/orgs/baselabs.png"}, "pr": 260, "new": true, "show_notes": true}. New in the 2026-10-18 PR batch (PR #244); updated to revision 9d1962f in PR #260. Public 0.3 run from PR #260 adopted (kit rescore 75.34, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving at revision 8f9f8619, whose weights are identical to 9d1962f; decision agreement with the submitted rows 99.85% on 831 rows (bar 98.5%). Weights are public on the Hugging Face Hub under a proprietary license. Training data not disclosed. One private new-domain test is not counted for this model: our checks found signs its training data overlapped it. Its new-domain score uses the remaining tests, put on the same scale with the same stock models. Decision Index 0.4.1 (release-v4), source generated 2026-10-29T14:04:16Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Source notes: New in the 2026-10-18 PR batch (PR #244); updated to revision 9d1962f in PR #260. Public 0.3 run from PR #260 adopted (kit rescore 75.34, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving at revision 8f9f8619, whose weights are identical to 9d1962f; decision agreement with the submitted rows 99.85% on 831 rows (bar 98.5%). Weights are public on the Hugging Face Hub under a proprietary license. Training data not disclosed. One private new-domain test is not counted for this model: our checks found signs its training data overlapped it. Its new-domain score uses the remaining tests, put on the same scale with the same stock models. Author public-run adoption/execution status and private lab setup are disclosed in configuration; evaluator-published does not mean every public run was independently executed. Published weighted contribution; not a distinct benchmark execution. The board served 9d1962f193; the registry pins the newest repository revision, and the commit titles after it concern the card. Decision Index 0.4 (release-v4), source generated 2026-10-20T14:13:56Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Checked / reported: 2026-11-02 · Samples: Not published Snapshot: 2026-10-20T14:13:56Z · Revision: 133313cf8963f2b6bed960abb0152fd14cbb6731; c130faf65f82e81d3bdefb01e758c712dc5c5199 for the 2 November 2026 additions · Published observations; result reuse terms not specified Result source · Benchmark source
HF text 0.4 · Full contribution · same_skills	39.59	Setup & source HF text 0.4 · Full contribution · same_skills Exact source engine mammoth-1; {"kind": "full fine-tune", "technique": "autoregressive", "technique_name": "autoregressive", "served_params": 27356728560, "params_estimated": false, "params_from_name": false, "trained_bytes": null, "model_url": "https://huggingface.co/baselabs/mammoth-1", "code_url": null, "base_model": "Qwen/Qwen3.5-27B", "base_url": "https://huggingface.co/Qwen/Qwen3.5-27B", "weights_repo": "baselabs/mammoth-1", "served_checkpoint": "9d1962f193e48a68cf0f3598dc7fb4462fd23ad9", "org": {"id": "baselabs", "name": "baselabs", "logo": "data/orgs/baselabs.png"}, "pr": 260, "new": true, "show_notes": true}. New in the 2026-10-18 PR batch (PR #244); updated to revision 9d1962f in PR #260. Public 0.3 run from PR #260 adopted (kit rescore 75.34, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving at revision 8f9f8619, whose weights are identical to 9d1962f; decision agreement with the submitted rows 99.85% on 831 rows (bar 98.5%). Weights are public on the Hugging Face Hub under a proprietary license. Training data not disclosed. One private new-domain test is not counted for this model: our checks found signs its training data overlapped it. Its new-domain score uses the remaining tests, put on the same scale with the same stock models. Decision Index 0.4.1 (release-v4), source generated 2026-10-29T14:04:16Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Source notes: New in the 2026-10-18 PR batch (PR #244); updated to revision 9d1962f in PR #260. Public 0.3 run from PR #260 adopted (kit rescore 75.34, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving at revision 8f9f8619, whose weights are identical to 9d1962f; decision agreement with the submitted rows 99.85% on 831 rows (bar 98.5%). Weights are public on the Hugging Face Hub under a proprietary license. Training data not disclosed. One private new-domain test is not counted for this model: our checks found signs its training data overlapped it. Its new-domain score uses the remaining tests, put on the same scale with the same stock models. Author public-run adoption/execution status and private lab setup are disclosed in configuration; evaluator-published does not mean every public run was independently executed. Published weighted contribution; not a distinct benchmark execution. The board served 9d1962f193; the registry pins the newest repository revision, and the commit titles after it concern the card. Decision Index 0.4 (release-v4), source generated 2026-10-20T14:13:56Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Checked / reported: 2026-11-02 · Samples: Not published Snapshot: 2026-10-20T14:13:56Z · Revision: 133313cf8963f2b6bed960abb0152fd14cbb6731; c130faf65f82e81d3bdefb01e758c712dc5c5199 for the 2 November 2026 additions · Published observations; result reuse terms not specified Result source · Benchmark source
HF text 0.4.1 · Full contribution · new_domains	28.86	Setup & source HF text 0.4.1 · Full contribution · new_domains Exact source engine mammoth-1; {"kind": "full fine-tune", "technique": "autoregressive", "technique_name": "autoregressive", "served_params": 27356728560, "params_estimated": false, "params_from_name": false, "trained_bytes": null, "model_url": "https://huggingface.co/baselabs/mammoth-1", "code_url": null, "base_model": "Qwen/Qwen3.5-27B", "base_url": "https://huggingface.co/Qwen/Qwen3.5-27B", "weights_repo": "baselabs/mammoth-1", "served_checkpoint": "9d1962f193e48a68cf0f3598dc7fb4462fd23ad9", "org": {"id": "baselabs", "name": "baselabs", "logo": "data/orgs/baselabs.png"}, "pr": 260, "new": true, "show_notes": true}. New in the 2026-10-18 PR batch (PR #244); updated to revision 9d1962f in PR #260. Public 0.3 run from PR #260 adopted (kit rescore 75.34, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving at revision 8f9f8619, whose weights are identical to 9d1962f; decision agreement with the submitted rows 99.85% on 831 rows (bar 98.5%). Weights are public on the Hugging Face Hub under a proprietary license. Training data not disclosed. One private new-domain test is not counted for this model: our checks found signs its training data overlapped it. Its new-domain score uses the remaining tests, put on the same scale with the same stock models. Decision Index 0.4.1 (release-v4), source generated 2026-10-29T14:04:16Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Source notes: New in the 2026-10-18 PR batch (PR #244); updated to revision 9d1962f in PR #260. Public 0.3 run from PR #260 adopted (kit rescore 75.34, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving at revision 8f9f8619, whose weights are identical to 9d1962f; decision agreement with the submitted rows 99.85% on 831 rows (bar 98.5%). Weights are public on the Hugging Face Hub under a proprietary license. Training data not disclosed. One private new-domain test is not counted for this model: our checks found signs its training data overlapped it. Its new-domain score uses the remaining tests, put on the same scale with the same stock models. Author public-run adoption/execution status and private lab setup are disclosed in configuration; evaluator-published does not mean every public run was independently executed. Published weighted contribution; not a distinct benchmark execution. The board served 9d1962f193; the registry pins the newest repository revision, and the commit titles after it concern the card. Community Decision Index 0.4.1: expanded/refreshed harder private new-domain tasks and equating; Full remains 60% private same-skill +40% private new domains, zero public contribution. This population revision is not a model upgrade/regression. Public/native/calibration data remain on their unchanged 0.4 definitions. Source publication/generation is not a model run date. Public submitted runs plus evaluator private measurements are Evaluator-published, not wholly independently executed. Checked / reported: 2026-11-02 · Samples: Not published Snapshot: 2026-10-22T05:40:06Z · Revision: d9962113a1acc95c2d7522c2babfb989afbeb034; c130faf65f82e81d3bdefb01e758c712dc5c5199 for the 2 November 2026 additions · Published observations; result reuse terms not specified Result source · Benchmark source
HF text 0.4 · Knowledge · raw	73.51	Setup & source HF text 0.4 · Knowledge · raw Exact source engine mammoth-1; {"kind": "full fine-tune", "technique": "autoregressive", "technique_name": "autoregressive", "served_params": 27356728560, "params_estimated": false, "params_from_name": false, "trained_bytes": null, "model_url": "https://huggingface.co/baselabs/mammoth-1", "code_url": null, "base_model": "Qwen/Qwen3.5-27B", "base_url": "https://huggingface.co/Qwen/Qwen3.5-27B", "weights_repo": "baselabs/mammoth-1", "served_checkpoint": "9d1962f193e48a68cf0f3598dc7fb4462fd23ad9", "org": {"id": "baselabs", "name": "baselabs", "logo": "data/orgs/baselabs.png"}, "pr": 260, "new": true, "show_notes": true}. New in the 2026-10-18 PR batch (PR #244); updated to revision 9d1962f in PR #260. Public 0.3 run from PR #260 adopted (kit rescore 75.34, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving at revision 8f9f8619, whose weights are identical to 9d1962f; decision agreement with the submitted rows 99.85% on 831 rows (bar 98.5%). Weights are public on the Hugging Face Hub under a proprietary license. Training data not disclosed. One private new-domain test is not counted for this model: our checks found signs its training data overlapped it. Its new-domain score uses the remaining tests, put on the same scale with the same stock models. Decision Index 0.4.1 (release-v4), source generated 2026-10-29T14:04:16Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Source notes: New in the 2026-10-18 PR batch (PR #244); updated to revision 9d1962f in PR #260. Public 0.3 run from PR #260 adopted (kit rescore 75.34, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving at revision 8f9f8619, whose weights are identical to 9d1962f; decision agreement with the submitted rows 99.85% on 831 rows (bar 98.5%). Weights are public on the Hugging Face Hub under a proprietary license. Training data not disclosed. One private new-domain test is not counted for this model: our checks found signs its training data overlapped it. Its new-domain score uses the remaining tests, put on the same scale with the same stock models. Author public-run adoption/execution status and private lab setup are disclosed in configuration; evaluator-published does not mean every public run was independently executed. Count is contributing benchmarks, not requests. Source area context: {"id": "knowledge", "raw": 0.7351, "skill": 0.6799, "combined": 0.521, "coverage": 1, "pending": 0, "n": 11} The board served 9d1962f193; the registry pins the newest repository revision, and the commit titles after it concern the card. Decision Index 0.4 (release-v4), source generated 2026-10-20T14:13:56Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Checked / reported: 2026-11-02 · Samples: 11 Snapshot: 2026-10-20T14:13:56Z · Revision: 133313cf8963f2b6bed960abb0152fd14cbb6731; c130faf65f82e81d3bdefb01e758c712dc5c5199 for the 2 November 2026 additions · Published observations; result reuse terms not specified Result source · Benchmark source
HF text 0.4 · Knowledge · skill	67.99	Setup & source HF text 0.4 · Knowledge · skill Exact source engine mammoth-1; {"kind": "full fine-tune", "technique": "autoregressive", "technique_name": "autoregressive", "served_params": 27356728560, "params_estimated": false, "params_from_name": false, "trained_bytes": null, "model_url": "https://huggingface.co/baselabs/mammoth-1", "code_url": null, "base_model": "Qwen/Qwen3.5-27B", "base_url": "https://huggingface.co/Qwen/Qwen3.5-27B", "weights_repo": "baselabs/mammoth-1", "served_checkpoint": "9d1962f193e48a68cf0f3598dc7fb4462fd23ad9", "org": {"id": "baselabs", "name": "baselabs", "logo": "data/orgs/baselabs.png"}, "pr": 260, "new": true, "show_notes": true}. New in the 2026-10-18 PR batch (PR #244); updated to revision 9d1962f in PR #260. Public 0.3 run from PR #260 adopted (kit rescore 75.34, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving at revision 8f9f8619, whose weights are identical to 9d1962f; decision agreement with the submitted rows 99.85% on 831 rows (bar 98.5%). Weights are public on the Hugging Face Hub under a proprietary license. Training data not disclosed. One private new-domain test is not counted for this model: our checks found signs its training data overlapped it. Its new-domain score uses the remaining tests, put on the same scale with the same stock models. Decision Index 0.4.1 (release-v4), source generated 2026-10-29T14:04:16Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Source notes: New in the 2026-10-18 PR batch (PR #244); updated to revision 9d1962f in PR #260. Public 0.3 run from PR #260 adopted (kit rescore 75.34, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving at revision 8f9f8619, whose weights are identical to 9d1962f; decision agreement with the submitted rows 99.85% on 831 rows (bar 98.5%). Weights are public on the Hugging Face Hub under a proprietary license. Training data not disclosed. One private new-domain test is not counted for this model: our checks found signs its training data overlapped it. Its new-domain score uses the remaining tests, put on the same scale with the same stock models. Author public-run adoption/execution status and private lab setup are disclosed in configuration; evaluator-published does not mean every public run was independently executed. Count is contributing benchmarks, not requests. Source area context: {"id": "knowledge", "raw": 0.7351, "skill": 0.6799, "combined": 0.521, "coverage": 1, "pending": 0, "n": 11} The board served 9d1962f193; the registry pins the newest repository revision, and the commit titles after it concern the card. Decision Index 0.4 (release-v4), source generated 2026-10-20T14:13:56Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Checked / reported: 2026-11-02 · Samples: 11 Snapshot: 2026-10-20T14:13:56Z · Revision: 133313cf8963f2b6bed960abb0152fd14cbb6731; c130faf65f82e81d3bdefb01e758c712dc5c5199 for the 2 November 2026 additions · Published observations; result reuse terms not specified Result source · Benchmark source
HF text 0.4 · Knowledge · combined	52.10	Setup & source HF text 0.4 · Knowledge · combined Exact source engine mammoth-1; {"kind": "full fine-tune", "technique": "autoregressive", "technique_name": "autoregressive", "served_params": 27356728560, "params_estimated": false, "params_from_name": false, "trained_bytes": null, "model_url": "https://huggingface.co/baselabs/mammoth-1", "code_url": null, "base_model": "Qwen/Qwen3.5-27B", "base_url": "https://huggingface.co/Qwen/Qwen3.5-27B", "weights_repo": "baselabs/mammoth-1", "served_checkpoint": "9d1962f193e48a68cf0f3598dc7fb4462fd23ad9", "org": {"id": "baselabs", "name": "baselabs", "logo": "data/orgs/baselabs.png"}, "pr": 260, "new": true, "show_notes": true}. New in the 2026-10-18 PR batch (PR #244); updated to revision 9d1962f in PR #260. Public 0.3 run from PR #260 adopted (kit rescore 75.34, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving at revision 8f9f8619, whose weights are identical to 9d1962f; decision agreement with the submitted rows 99.85% on 831 rows (bar 98.5%). Weights are public on the Hugging Face Hub under a proprietary license. Training data not disclosed. One private new-domain test is not counted for this model: our checks found signs its training data overlapped it. Its new-domain score uses the remaining tests, put on the same scale with the same stock models. Decision Index 0.4.1 (release-v4), source generated 2026-10-29T14:04:16Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Source notes: New in the 2026-10-18 PR batch (PR #244); updated to revision 9d1962f in PR #260. Public 0.3 run from PR #260 adopted (kit rescore 75.34, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving at revision 8f9f8619, whose weights are identical to 9d1962f; decision agreement with the submitted rows 99.85% on 831 rows (bar 98.5%). Weights are public on the Hugging Face Hub under a proprietary license. Training data not disclosed. One private new-domain test is not counted for this model: our checks found signs its training data overlapped it. Its new-domain score uses the remaining tests, put on the same scale with the same stock models. Author public-run adoption/execution status and private lab setup are disclosed in configuration; evaluator-published does not mean every public run was independently executed. Count is contributing benchmarks, not requests. Source area context: {"id": "knowledge", "raw": 0.7351, "skill": 0.6799, "combined": 0.521, "coverage": 1, "pending": 0, "n": 11} The board served 9d1962f193; the registry pins the newest repository revision, and the commit titles after it concern the card. Decision Index 0.4 (release-v4), source generated 2026-10-20T14:13:56Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Checked / reported: 2026-11-02 · Samples: 11 Snapshot: 2026-10-20T14:13:56Z · Revision: 133313cf8963f2b6bed960abb0152fd14cbb6731; c130faf65f82e81d3bdefb01e758c712dc5c5199 for the 2 November 2026 additions · Published observations; result reuse terms not specified Result source · Benchmark source
HF text 0.4 · Language · raw	83.40	Setup & source HF text 0.4 · Language · raw Exact source engine mammoth-1; {"kind": "full fine-tune", "technique": "autoregressive", "technique_name": "autoregressive", "served_params": 27356728560, "params_estimated": false, "params_from_name": false, "trained_bytes": null, "model_url": "https://huggingface.co/baselabs/mammoth-1", "code_url": null, "base_model": "Qwen/Qwen3.5-27B", "base_url": "https://huggingface.co/Qwen/Qwen3.5-27B", "weights_repo": "baselabs/mammoth-1", "served_checkpoint": "9d1962f193e48a68cf0f3598dc7fb4462fd23ad9", "org": {"id": "baselabs", "name": "baselabs", "logo": "data/orgs/baselabs.png"}, "pr": 260, "new": true, "show_notes": true}. New in the 2026-10-18 PR batch (PR #244); updated to revision 9d1962f in PR #260. Public 0.3 run from PR #260 adopted (kit rescore 75.34, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving at revision 8f9f8619, whose weights are identical to 9d1962f; decision agreement with the submitted rows 99.85% on 831 rows (bar 98.5%). Weights are public on the Hugging Face Hub under a proprietary license. Training data not disclosed. One private new-domain test is not counted for this model: our checks found signs its training data overlapped it. Its new-domain score uses the remaining tests, put on the same scale with the same stock models. Decision Index 0.4.1 (release-v4), source generated 2026-10-29T14:04:16Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Source notes: New in the 2026-10-18 PR batch (PR #244); updated to revision 9d1962f in PR #260. Public 0.3 run from PR #260 adopted (kit rescore 75.34, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving at revision 8f9f8619, whose weights are identical to 9d1962f; decision agreement with the submitted rows 99.85% on 831 rows (bar 98.5%). Weights are public on the Hugging Face Hub under a proprietary license. Training data not disclosed. One private new-domain test is not counted for this model: our checks found signs its training data overlapped it. Its new-domain score uses the remaining tests, put on the same scale with the same stock models. Author public-run adoption/execution status and private lab setup are disclosed in configuration; evaluator-published does not mean every public run was independently executed. Count is contributing benchmarks, not requests. Source area context: {"id": "language", "raw": 0.834, "skill": 0.7611, "combined": 0.6709, "coverage": 1, "pending": 0, "n": 9} The board served 9d1962f193; the registry pins the newest repository revision, and the commit titles after it concern the card. Decision Index 0.4 (release-v4), source generated 2026-10-20T14:13:56Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Checked / reported: 2026-11-02 · Samples: 9 Snapshot: 2026-10-20T14:13:56Z · Revision: 133313cf8963f2b6bed960abb0152fd14cbb6731; c130faf65f82e81d3bdefb01e758c712dc5c5199 for the 2 November 2026 additions · Published observations; result reuse terms not specified Result source · Benchmark source
HF text 0.4 · Language · skill	76.11	Setup & source HF text 0.4 · Language · skill Exact source engine mammoth-1; {"kind": "full fine-tune", "technique": "autoregressive", "technique_name": "autoregressive", "served_params": 27356728560, "params_estimated": false, "params_from_name": false, "trained_bytes": null, "model_url": "https://huggingface.co/baselabs/mammoth-1", "code_url": null, "base_model": "Qwen/Qwen3.5-27B", "base_url": "https://huggingface.co/Qwen/Qwen3.5-27B", "weights_repo": "baselabs/mammoth-1", "served_checkpoint": "9d1962f193e48a68cf0f3598dc7fb4462fd23ad9", "org": {"id": "baselabs", "name": "baselabs", "logo": "data/orgs/baselabs.png"}, "pr": 260, "new": true, "show_notes": true}. New in the 2026-10-18 PR batch (PR #244); updated to revision 9d1962f in PR #260. Public 0.3 run from PR #260 adopted (kit rescore 75.34, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving at revision 8f9f8619, whose weights are identical to 9d1962f; decision agreement with the submitted rows 99.85% on 831 rows (bar 98.5%). Weights are public on the Hugging Face Hub under a proprietary license. Training data not disclosed. One private new-domain test is not counted for this model: our checks found signs its training data overlapped it. Its new-domain score uses the remaining tests, put on the same scale with the same stock models. Decision Index 0.4.1 (release-v4), source generated 2026-10-29T14:04:16Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Source notes: New in the 2026-10-18 PR batch (PR #244); updated to revision 9d1962f in PR #260. Public 0.3 run from PR #260 adopted (kit rescore 75.34, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving at revision 8f9f8619, whose weights are identical to 9d1962f; decision agreement with the submitted rows 99.85% on 831 rows (bar 98.5%). Weights are public on the Hugging Face Hub under a proprietary license. Training data not disclosed. One private new-domain test is not counted for this model: our checks found signs its training data overlapped it. Its new-domain score uses the remaining tests, put on the same scale with the same stock models. Author public-run adoption/execution status and private lab setup are disclosed in configuration; evaluator-published does not mean every public run was independently executed. Count is contributing benchmarks, not requests. Source area context: {"id": "language", "raw": 0.834, "skill": 0.7611, "combined": 0.6709, "coverage": 1, "pending": 0, "n": 9} The board served 9d1962f193; the registry pins the newest repository revision, and the commit titles after it concern the card. Decision Index 0.4 (release-v4), source generated 2026-10-20T14:13:56Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Checked / reported: 2026-11-02 · Samples: 9 Snapshot: 2026-10-20T14:13:56Z · Revision: 133313cf8963f2b6bed960abb0152fd14cbb6731; c130faf65f82e81d3bdefb01e758c712dc5c5199 for the 2 November 2026 additions · Published observations; result reuse terms not specified Result source · Benchmark source
HF text 0.4 · Language · combined	67.09	Setup & source HF text 0.4 · Language · combined Exact source engine mammoth-1; {"kind": "full fine-tune", "technique": "autoregressive", "technique_name": "autoregressive", "served_params": 27356728560, "params_estimated": false, "params_from_name": false, "trained_bytes": null, "model_url": "https://huggingface.co/baselabs/mammoth-1", "code_url": null, "base_model": "Qwen/Qwen3.5-27B", "base_url": "https://huggingface.co/Qwen/Qwen3.5-27B", "weights_repo": "baselabs/mammoth-1", "served_checkpoint": "9d1962f193e48a68cf0f3598dc7fb4462fd23ad9", "org": {"id": "baselabs", "name": "baselabs", "logo": "data/orgs/baselabs.png"}, "pr": 260, "new": true, "show_notes": true}. New in the 2026-10-18 PR batch (PR #244); updated to revision 9d1962f in PR #260. Public 0.3 run from PR #260 adopted (kit rescore 75.34, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving at revision 8f9f8619, whose weights are identical to 9d1962f; decision agreement with the submitted rows 99.85% on 831 rows (bar 98.5%). Weights are public on the Hugging Face Hub under a proprietary license. Training data not disclosed. One private new-domain test is not counted for this model: our checks found signs its training data overlapped it. Its new-domain score uses the remaining tests, put on the same scale with the same stock models. Decision Index 0.4.1 (release-v4), source generated 2026-10-29T14:04:16Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Source notes: New in the 2026-10-18 PR batch (PR #244); updated to revision 9d1962f in PR #260. Public 0.3 run from PR #260 adopted (kit rescore 75.34, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving at revision 8f9f8619, whose weights are identical to 9d1962f; decision agreement with the submitted rows 99.85% on 831 rows (bar 98.5%). Weights are public on the Hugging Face Hub under a proprietary license. Training data not disclosed. One private new-domain test is not counted for this model: our checks found signs its training data overlapped it. Its new-domain score uses the remaining tests, put on the same scale with the same stock models. Author public-run adoption/execution status and private lab setup are disclosed in configuration; evaluator-published does not mean every public run was independently executed. Count is contributing benchmarks, not requests. Source area context: {"id": "language", "raw": 0.834, "skill": 0.7611, "combined": 0.6709, "coverage": 1, "pending": 0, "n": 9} The board served 9d1962f193; the registry pins the newest repository revision, and the commit titles after it concern the card. Decision Index 0.4 (release-v4), source generated 2026-10-20T14:13:56Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Checked / reported: 2026-11-02 · Samples: 9 Snapshot: 2026-10-20T14:13:56Z · Revision: 133313cf8963f2b6bed960abb0152fd14cbb6731; c130faf65f82e81d3bdefb01e758c712dc5c5199 for the 2 November 2026 additions · Published observations; result reuse terms not specified Result source · Benchmark source
HF text 0.4 · Retrieval · raw	81.82	Setup & source HF text 0.4 · Retrieval · raw Exact source engine mammoth-1; {"kind": "full fine-tune", "technique": "autoregressive", "technique_name": "autoregressive", "served_params": 27356728560, "params_estimated": false, "params_from_name": false, "trained_bytes": null, "model_url": "https://huggingface.co/baselabs/mammoth-1", "code_url": null, "base_model": "Qwen/Qwen3.5-27B", "base_url": "https://huggingface.co/Qwen/Qwen3.5-27B", "weights_repo": "baselabs/mammoth-1", "served_checkpoint": "9d1962f193e48a68cf0f3598dc7fb4462fd23ad9", "org": {"id": "baselabs", "name": "baselabs", "logo": "data/orgs/baselabs.png"}, "pr": 260, "new": true, "show_notes": true}. New in the 2026-10-18 PR batch (PR #244); updated to revision 9d1962f in PR #260. Public 0.3 run from PR #260 adopted (kit rescore 75.34, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving at revision 8f9f8619, whose weights are identical to 9d1962f; decision agreement with the submitted rows 99.85% on 831 rows (bar 98.5%). Weights are public on the Hugging Face Hub under a proprietary license. Training data not disclosed. One private new-domain test is not counted for this model: our checks found signs its training data overlapped it. Its new-domain score uses the remaining tests, put on the same scale with the same stock models. Decision Index 0.4.1 (release-v4), source generated 2026-10-29T14:04:16Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Source notes: New in the 2026-10-18 PR batch (PR #244); updated to revision 9d1962f in PR #260. Public 0.3 run from PR #260 adopted (kit rescore 75.34, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving at revision 8f9f8619, whose weights are identical to 9d1962f; decision agreement with the submitted rows 99.85% on 831 rows (bar 98.5%). Weights are public on the Hugging Face Hub under a proprietary license. Training data not disclosed. One private new-domain test is not counted for this model: our checks found signs its training data overlapped it. Its new-domain score uses the remaining tests, put on the same scale with the same stock models. Author public-run adoption/execution status and private lab setup are disclosed in configuration; evaluator-published does not mean every public run was independently executed. Count is contributing benchmarks, not requests. Source area context: {"id": "retrieval", "raw": 0.8182, "skill": 0.7737, "combined": 0.6624, "coverage": 1, "pending": 0, "n": 6} The board served 9d1962f193; the registry pins the newest repository revision, and the commit titles after it concern the card. Decision Index 0.4 (release-v4), source generated 2026-10-20T14:13:56Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Checked / reported: 2026-11-02 · Samples: 6 Snapshot: 2026-10-20T14:13:56Z · Revision: 133313cf8963f2b6bed960abb0152fd14cbb6731; c130faf65f82e81d3bdefb01e758c712dc5c5199 for the 2 November 2026 additions · Published observations; result reuse terms not specified Result source · Benchmark source
HF text 0.4 · Retrieval · skill	77.37	Setup & source HF text 0.4 · Retrieval · skill Exact source engine mammoth-1; {"kind": "full fine-tune", "technique": "autoregressive", "technique_name": "autoregressive", "served_params": 27356728560, "params_estimated": false, "params_from_name": false, "trained_bytes": null, "model_url": "https://huggingface.co/baselabs/mammoth-1", "code_url": null, "base_model": "Qwen/Qwen3.5-27B", "base_url": "https://huggingface.co/Qwen/Qwen3.5-27B", "weights_repo": "baselabs/mammoth-1", "served_checkpoint": "9d1962f193e48a68cf0f3598dc7fb4462fd23ad9", "org": {"id": "baselabs", "name": "baselabs", "logo": "data/orgs/baselabs.png"}, "pr": 260, "new": true, "show_notes": true}. New in the 2026-10-18 PR batch (PR #244); updated to revision 9d1962f in PR #260. Public 0.3 run from PR #260 adopted (kit rescore 75.34, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving at revision 8f9f8619, whose weights are identical to 9d1962f; decision agreement with the submitted rows 99.85% on 831 rows (bar 98.5%). Weights are public on the Hugging Face Hub under a proprietary license. Training data not disclosed. One private new-domain test is not counted for this model: our checks found signs its training data overlapped it. Its new-domain score uses the remaining tests, put on the same scale with the same stock models. Decision Index 0.4.1 (release-v4), source generated 2026-10-29T14:04:16Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Source notes: New in the 2026-10-18 PR batch (PR #244); updated to revision 9d1962f in PR #260. Public 0.3 run from PR #260 adopted (kit rescore 75.34, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving at revision 8f9f8619, whose weights are identical to 9d1962f; decision agreement with the submitted rows 99.85% on 831 rows (bar 98.5%). Weights are public on the Hugging Face Hub under a proprietary license. Training data not disclosed. One private new-domain test is not counted for this model: our checks found signs its training data overlapped it. Its new-domain score uses the remaining tests, put on the same scale with the same stock models. Author public-run adoption/execution status and private lab setup are disclosed in configuration; evaluator-published does not mean every public run was independently executed. Count is contributing benchmarks, not requests. Source area context: {"id": "retrieval", "raw": 0.8182, "skill": 0.7737, "combined": 0.6624, "coverage": 1, "pending": 0, "n": 6} The board served 9d1962f193; the registry pins the newest repository revision, and the commit titles after it concern the card. Decision Index 0.4 (release-v4), source generated 2026-10-20T14:13:56Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Checked / reported: 2026-11-02 · Samples: 6 Snapshot: 2026-10-20T14:13:56Z · Revision: 133313cf8963f2b6bed960abb0152fd14cbb6731; c130faf65f82e81d3bdefb01e758c712dc5c5199 for the 2 November 2026 additions · Published observations; result reuse terms not specified Result source · Benchmark source
HF text 0.4 · Retrieval · combined	66.24	Setup & source HF text 0.4 · Retrieval · combined Exact source engine mammoth-1; {"kind": "full fine-tune", "technique": "autoregressive", "technique_name": "autoregressive", "served_params": 27356728560, "params_estimated": false, "params_from_name": false, "trained_bytes": null, "model_url": "https://huggingface.co/baselabs/mammoth-1", "code_url": null, "base_model": "Qwen/Qwen3.5-27B", "base_url": "https://huggingface.co/Qwen/Qwen3.5-27B", "weights_repo": "baselabs/mammoth-1", "served_checkpoint": "9d1962f193e48a68cf0f3598dc7fb4462fd23ad9", "org": {"id": "baselabs", "name": "baselabs", "logo": "data/orgs/baselabs.png"}, "pr": 260, "new": true, "show_notes": true}. New in the 2026-10-18 PR batch (PR #244); updated to revision 9d1962f in PR #260. Public 0.3 run from PR #260 adopted (kit rescore 75.34, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving at revision 8f9f8619, whose weights are identical to 9d1962f; decision agreement with the submitted rows 99.85% on 831 rows (bar 98.5%). Weights are public on the Hugging Face Hub under a proprietary license. Training data not disclosed. One private new-domain test is not counted for this model: our checks found signs its training data overlapped it. Its new-domain score uses the remaining tests, put on the same scale with the same stock models. Decision Index 0.4.1 (release-v4), source generated 2026-10-29T14:04:16Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Source notes: New in the 2026-10-18 PR batch (PR #244); updated to revision 9d1962f in PR #260. Public 0.3 run from PR #260 adopted (kit rescore 75.34, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving at revision 8f9f8619, whose weights are identical to 9d1962f; decision agreement with the submitted rows 99.85% on 831 rows (bar 98.5%). Weights are public on the Hugging Face Hub under a proprietary license. Training data not disclosed. One private new-domain test is not counted for this model: our checks found signs its training data overlapped it. Its new-domain score uses the remaining tests, put on the same scale with the same stock models. Author public-run adoption/execution status and private lab setup are disclosed in configuration; evaluator-published does not mean every public run was independently executed. Count is contributing benchmarks, not requests. Source area context: {"id": "retrieval", "raw": 0.8182, "skill": 0.7737, "combined": 0.6624, "coverage": 1, "pending": 0, "n": 6} The board served 9d1962f193; the registry pins the newest repository revision, and the commit titles after it concern the card. Decision Index 0.4 (release-v4), source generated 2026-10-20T14:13:56Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Checked / reported: 2026-11-02 · Samples: 6 Snapshot: 2026-10-20T14:13:56Z · Revision: 133313cf8963f2b6bed960abb0152fd14cbb6731; c130faf65f82e81d3bdefb01e758c712dc5c5199 for the 2 November 2026 additions · Published observations; result reuse terms not specified Result source · Benchmark source
HF text 0.4 · Tools · raw	90.14	Setup & source HF text 0.4 · Tools · raw Exact source engine mammoth-1; {"kind": "full fine-tune", "technique": "autoregressive", "technique_name": "autoregressive", "served_params": 27356728560, "params_estimated": false, "params_from_name": false, "trained_bytes": null, "model_url": "https://huggingface.co/baselabs/mammoth-1", "code_url": null, "base_model": "Qwen/Qwen3.5-27B", "base_url": "https://huggingface.co/Qwen/Qwen3.5-27B", "weights_repo": "baselabs/mammoth-1", "served_checkpoint": "9d1962f193e48a68cf0f3598dc7fb4462fd23ad9", "org": {"id": "baselabs", "name": "baselabs", "logo": "data/orgs/baselabs.png"}, "pr": 260, "new": true, "show_notes": true}. New in the 2026-10-18 PR batch (PR #244); updated to revision 9d1962f in PR #260. Public 0.3 run from PR #260 adopted (kit rescore 75.34, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving at revision 8f9f8619, whose weights are identical to 9d1962f; decision agreement with the submitted rows 99.85% on 831 rows (bar 98.5%). Weights are public on the Hugging Face Hub under a proprietary license. Training data not disclosed. One private new-domain test is not counted for this model: our checks found signs its training data overlapped it. Its new-domain score uses the remaining tests, put on the same scale with the same stock models. Decision Index 0.4.1 (release-v4), source generated 2026-10-29T14:04:16Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Source notes: New in the 2026-10-18 PR batch (PR #244); updated to revision 9d1962f in PR #260. Public 0.3 run from PR #260 adopted (kit rescore 75.34, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving at revision 8f9f8619, whose weights are identical to 9d1962f; decision agreement with the submitted rows 99.85% on 831 rows (bar 98.5%). Weights are public on the Hugging Face Hub under a proprietary license. Training data not disclosed. One private new-domain test is not counted for this model: our checks found signs its training data overlapped it. Its new-domain score uses the remaining tests, put on the same scale with the same stock models. Author public-run adoption/execution status and private lab setup are disclosed in configuration; evaluator-published does not mean every public run was independently executed. Count is contributing benchmarks, not requests. Source area context: {"id": "tools", "raw": 0.9014, "skill": 0.886, "combined": 0.8343, "coverage": 1, "pending": 0, "n": 5} The board served 9d1962f193; the registry pins the newest repository revision, and the commit titles after it concern the card. Decision Index 0.4 (release-v4), source generated 2026-10-20T14:13:56Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Checked / reported: 2026-11-02 · Samples: 5 Snapshot: 2026-10-20T14:13:56Z · Revision: 133313cf8963f2b6bed960abb0152fd14cbb6731; c130faf65f82e81d3bdefb01e758c712dc5c5199 for the 2 November 2026 additions · Published observations; result reuse terms not specified Result source · Benchmark source
HF text 0.4 · Tools · skill	88.60	Setup & source HF text 0.4 · Tools · skill Exact source engine mammoth-1; {"kind": "full fine-tune", "technique": "autoregressive", "technique_name": "autoregressive", "served_params": 27356728560, "params_estimated": false, "params_from_name": false, "trained_bytes": null, "model_url": "https://huggingface.co/baselabs/mammoth-1", "code_url": null, "base_model": "Qwen/Qwen3.5-27B", "base_url": "https://huggingface.co/Qwen/Qwen3.5-27B", "weights_repo": "baselabs/mammoth-1", "served_checkpoint": "9d1962f193e48a68cf0f3598dc7fb4462fd23ad9", "org": {"id": "baselabs", "name": "baselabs", "logo": "data/orgs/baselabs.png"}, "pr": 260, "new": true, "show_notes": true}. New in the 2026-10-18 PR batch (PR #244); updated to revision 9d1962f in PR #260. Public 0.3 run from PR #260 adopted (kit rescore 75.34, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving at revision 8f9f8619, whose weights are identical to 9d1962f; decision agreement with the submitted rows 99.85% on 831 rows (bar 98.5%). Weights are public on the Hugging Face Hub under a proprietary license. Training data not disclosed. One private new-domain test is not counted for this model: our checks found signs its training data overlapped it. Its new-domain score uses the remaining tests, put on the same scale with the same stock models. Decision Index 0.4.1 (release-v4), source generated 2026-10-29T14:04:16Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Source notes: New in the 2026-10-18 PR batch (PR #244); updated to revision 9d1962f in PR #260. Public 0.3 run from PR #260 adopted (kit rescore 75.34, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving at revision 8f9f8619, whose weights are identical to 9d1962f; decision agreement with the submitted rows 99.85% on 831 rows (bar 98.5%). Weights are public on the Hugging Face Hub under a proprietary license. Training data not disclosed. One private new-domain test is not counted for this model: our checks found signs its training data overlapped it. Its new-domain score uses the remaining tests, put on the same scale with the same stock models. Author public-run adoption/execution status and private lab setup are disclosed in configuration; evaluator-published does not mean every public run was independently executed. Count is contributing benchmarks, not requests. Source area context: {"id": "tools", "raw": 0.9014, "skill": 0.886, "combined": 0.8343, "coverage": 1, "pending": 0, "n": 5} The board served 9d1962f193; the registry pins the newest repository revision, and the commit titles after it concern the card. Decision Index 0.4 (release-v4), source generated 2026-10-20T14:13:56Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Checked / reported: 2026-11-02 · Samples: 5 Snapshot: 2026-10-20T14:13:56Z · Revision: 133313cf8963f2b6bed960abb0152fd14cbb6731; c130faf65f82e81d3bdefb01e758c712dc5c5199 for the 2 November 2026 additions · Published observations; result reuse terms not specified Result source · Benchmark source
HF text 0.4 · Tools · combined	83.43	Setup & source HF text 0.4 · Tools · combined Exact source engine mammoth-1; {"kind": "full fine-tune", "technique": "autoregressive", "technique_name": "autoregressive", "served_params": 27356728560, "params_estimated": false, "params_from_name": false, "trained_bytes": null, "model_url": "https://huggingface.co/baselabs/mammoth-1", "code_url": null, "base_model": "Qwen/Qwen3.5-27B", "base_url": "https://huggingface.co/Qwen/Qwen3.5-27B", "weights_repo": "baselabs/mammoth-1", "served_checkpoint": "9d1962f193e48a68cf0f3598dc7fb4462fd23ad9", "org": {"id": "baselabs", "name": "baselabs", "logo": "data/orgs/baselabs.png"}, "pr": 260, "new": true, "show_notes": true}. New in the 2026-10-18 PR batch (PR #244); updated to revision 9d1962f in PR #260. Public 0.3 run from PR #260 adopted (kit rescore 75.34, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving at revision 8f9f8619, whose weights are identical to 9d1962f; decision agreement with the submitted rows 99.85% on 831 rows (bar 98.5%). Weights are public on the Hugging Face Hub under a proprietary license. Training data not disclosed. One private new-domain test is not counted for this model: our checks found signs its training data overlapped it. Its new-domain score uses the remaining tests, put on the same scale with the same stock models. Decision Index 0.4.1 (release-v4), source generated 2026-10-29T14:04:16Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Source notes: New in the 2026-10-18 PR batch (PR #244); updated to revision 9d1962f in PR #260. Public 0.3 run from PR #260 adopted (kit rescore 75.34, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving at revision 8f9f8619, whose weights are identical to 9d1962f; decision agreement with the submitted rows 99.85% on 831 rows (bar 98.5%). Weights are public on the Hugging Face Hub under a proprietary license. Training data not disclosed. One private new-domain test is not counted for this model: our checks found signs its training data overlapped it. Its new-domain score uses the remaining tests, put on the same scale with the same stock models. Author public-run adoption/execution status and private lab setup are disclosed in configuration; evaluator-published does not mean every public run was independently executed. Count is contributing benchmarks, not requests. Source area context: {"id": "tools", "raw": 0.9014, "skill": 0.886, "combined": 0.8343, "coverage": 1, "pending": 0, "n": 5} The board served 9d1962f193; the registry pins the newest repository revision, and the commit titles after it concern the card. Decision Index 0.4 (release-v4), source generated 2026-10-20T14:13:56Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Checked / reported: 2026-11-02 · Samples: 5 Snapshot: 2026-10-20T14:13:56Z · Revision: 133313cf8963f2b6bed960abb0152fd14cbb6731; c130faf65f82e81d3bdefb01e758c712dc5c5199 for the 2 November 2026 additions · Published observations; result reuse terms not specified Result source · Benchmark source
HF text 0.4 · Arts · raw	77.71	Setup & source HF text 0.4 · Arts · raw Exact source engine mammoth-1; {"kind": "full fine-tune", "technique": "autoregressive", "technique_name": "autoregressive", "served_params": 27356728560, "params_estimated": false, "params_from_name": false, "trained_bytes": null, "model_url": "https://huggingface.co/baselabs/mammoth-1", "code_url": null, "base_model": "Qwen/Qwen3.5-27B", "base_url": "https://huggingface.co/Qwen/Qwen3.5-27B", "weights_repo": "baselabs/mammoth-1", "served_checkpoint": "9d1962f193e48a68cf0f3598dc7fb4462fd23ad9", "org": {"id": "baselabs", "name": "baselabs", "logo": "data/orgs/baselabs.png"}, "pr": 260, "new": true, "show_notes": true}. New in the 2026-10-18 PR batch (PR #244); updated to revision 9d1962f in PR #260. Public 0.3 run from PR #260 adopted (kit rescore 75.34, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving at revision 8f9f8619, whose weights are identical to 9d1962f; decision agreement with the submitted rows 99.85% on 831 rows (bar 98.5%). Weights are public on the Hugging Face Hub under a proprietary license. Training data not disclosed. One private new-domain test is not counted for this model: our checks found signs its training data overlapped it. Its new-domain score uses the remaining tests, put on the same scale with the same stock models. Decision Index 0.4.1 (release-v4), source generated 2026-10-29T14:04:16Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Source notes: New in the 2026-10-18 PR batch (PR #244); updated to revision 9d1962f in PR #260. Public 0.3 run from PR #260 adopted (kit rescore 75.34, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving at revision 8f9f8619, whose weights are identical to 9d1962f; decision agreement with the submitted rows 99.85% on 831 rows (bar 98.5%). Weights are public on the Hugging Face Hub under a proprietary license. Training data not disclosed. One private new-domain test is not counted for this model: our checks found signs its training data overlapped it. Its new-domain score uses the remaining tests, put on the same scale with the same stock models. Author public-run adoption/execution status and private lab setup are disclosed in configuration; evaluator-published does not mean every public run was independently executed. Count is contributing benchmarks, not requests. Source area context: {"id": "arts", "raw": 0.7771, "skill": 0.6408, "combined": 0.627, "coverage": 1, "pending": 0, "n": 6} The board served 9d1962f193; the registry pins the newest repository revision, and the commit titles after it concern the card. Decision Index 0.4 (release-v4), source generated 2026-10-20T14:13:56Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Checked / reported: 2026-11-02 · Samples: 6 Snapshot: 2026-10-20T14:13:56Z · Revision: 133313cf8963f2b6bed960abb0152fd14cbb6731; c130faf65f82e81d3bdefb01e758c712dc5c5199 for the 2 November 2026 additions · Published observations; result reuse terms not specified Result source · Benchmark source
HF text 0.4 · Arts · skill	64.08	Setup & source HF text 0.4 · Arts · skill Exact source engine mammoth-1; {"kind": "full fine-tune", "technique": "autoregressive", "technique_name": "autoregressive", "served_params": 27356728560, "params_estimated": false, "params_from_name": false, "trained_bytes": null, "model_url": "https://huggingface.co/baselabs/mammoth-1", "code_url": null, "base_model": "Qwen/Qwen3.5-27B", "base_url": "https://huggingface.co/Qwen/Qwen3.5-27B", "weights_repo": "baselabs/mammoth-1", "served_checkpoint": "9d1962f193e48a68cf0f3598dc7fb4462fd23ad9", "org": {"id": "baselabs", "name": "baselabs", "logo": "data/orgs/baselabs.png"}, "pr": 260, "new": true, "show_notes": true}. New in the 2026-10-18 PR batch (PR #244); updated to revision 9d1962f in PR #260. Public 0.3 run from PR #260 adopted (kit rescore 75.34, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving at revision 8f9f8619, whose weights are identical to 9d1962f; decision agreement with the submitted rows 99.85% on 831 rows (bar 98.5%). Weights are public on the Hugging Face Hub under a proprietary license. Training data not disclosed. One private new-domain test is not counted for this model: our checks found signs its training data overlapped it. Its new-domain score uses the remaining tests, put on the same scale with the same stock models. Decision Index 0.4.1 (release-v4), source generated 2026-10-29T14:04:16Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Source notes: New in the 2026-10-18 PR batch (PR #244); updated to revision 9d1962f in PR #260. Public 0.3 run from PR #260 adopted (kit rescore 75.34, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving at revision 8f9f8619, whose weights are identical to 9d1962f; decision agreement with the submitted rows 99.85% on 831 rows (bar 98.5%). Weights are public on the Hugging Face Hub under a proprietary license. Training data not disclosed. One private new-domain test is not counted for this model: our checks found signs its training data overlapped it. Its new-domain score uses the remaining tests, put on the same scale with the same stock models. Author public-run adoption/execution status and private lab setup are disclosed in configuration; evaluator-published does not mean every public run was independently executed. Count is contributing benchmarks, not requests. Source area context: {"id": "arts", "raw": 0.7771, "skill": 0.6408, "combined": 0.627, "coverage": 1, "pending": 0, "n": 6} The board served 9d1962f193; the registry pins the newest repository revision, and the commit titles after it concern the card. Decision Index 0.4 (release-v4), source generated 2026-10-20T14:13:56Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Checked / reported: 2026-11-02 · Samples: 6 Snapshot: 2026-10-20T14:13:56Z · Revision: 133313cf8963f2b6bed960abb0152fd14cbb6731; c130faf65f82e81d3bdefb01e758c712dc5c5199 for the 2 November 2026 additions · Published observations; result reuse terms not specified Result source · Benchmark source
HF text 0.4 · Arts · combined	62.70	Setup & source HF text 0.4 · Arts · combined Exact source engine mammoth-1; {"kind": "full fine-tune", "technique": "autoregressive", "technique_name": "autoregressive", "served_params": 27356728560, "params_estimated": false, "params_from_name": false, "trained_bytes": null, "model_url": "https://huggingface.co/baselabs/mammoth-1", "code_url": null, "base_model": "Qwen/Qwen3.5-27B", "base_url": "https://huggingface.co/Qwen/Qwen3.5-27B", "weights_repo": "baselabs/mammoth-1", "served_checkpoint": "9d1962f193e48a68cf0f3598dc7fb4462fd23ad9", "org": {"id": "baselabs", "name": "baselabs", "logo": "data/orgs/baselabs.png"}, "pr": 260, "new": true, "show_notes": true}. New in the 2026-10-18 PR batch (PR #244); updated to revision 9d1962f in PR #260. Public 0.3 run from PR #260 adopted (kit rescore 75.34, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving at revision 8f9f8619, whose weights are identical to 9d1962f; decision agreement with the submitted rows 99.85% on 831 rows (bar 98.5%). Weights are public on the Hugging Face Hub under a proprietary license. Training data not disclosed. One private new-domain test is not counted for this model: our checks found signs its training data overlapped it. Its new-domain score uses the remaining tests, put on the same scale with the same stock models. Decision Index 0.4.1 (release-v4), source generated 2026-10-29T14:04:16Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Source notes: New in the 2026-10-18 PR batch (PR #244); updated to revision 9d1962f in PR #260. Public 0.3 run from PR #260 adopted (kit rescore 75.34, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving at revision 8f9f8619, whose weights are identical to 9d1962f; decision agreement with the submitted rows 99.85% on 831 rows (bar 98.5%). Weights are public on the Hugging Face Hub under a proprietary license. Training data not disclosed. One private new-domain test is not counted for this model: our checks found signs its training data overlapped it. Its new-domain score uses the remaining tests, put on the same scale with the same stock models. Author public-run adoption/execution status and private lab setup are disclosed in configuration; evaluator-published does not mean every public run was independently executed. Count is contributing benchmarks, not requests. Source area context: {"id": "arts", "raw": 0.7771, "skill": 0.6408, "combined": 0.627, "coverage": 1, "pending": 0, "n": 6} The board served 9d1962f193; the registry pins the newest repository revision, and the commit titles after it concern the card. Decision Index 0.4 (release-v4), source generated 2026-10-20T14:13:56Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Checked / reported: 2026-11-02 · Samples: 6 Snapshot: 2026-10-20T14:13:56Z · Revision: 133313cf8963f2b6bed960abb0152fd14cbb6731; c130faf65f82e81d3bdefb01e758c712dc5c5199 for the 2 November 2026 additions · Published observations; result reuse terms not specified Result source · Benchmark source
HF text calibration 0.4 · ECE	4.08 pp	Setup & source HF text calibration 0.4 · ECE Exact source engine mammoth-1; {"kind": "full fine-tune", "technique": "autoregressive", "technique_name": "autoregressive", "served_params": 27356728560, "params_estimated": false, "params_from_name": false, "trained_bytes": null, "model_url": "https://huggingface.co/baselabs/mammoth-1", "code_url": null, "base_model": "Qwen/Qwen3.5-27B", "base_url": "https://huggingface.co/Qwen/Qwen3.5-27B", "weights_repo": "baselabs/mammoth-1", "served_checkpoint": "9d1962f193e48a68cf0f3598dc7fb4462fd23ad9", "org": {"id": "baselabs", "name": "baselabs", "logo": "data/orgs/baselabs.png"}, "pr": 260, "new": true, "show_notes": true}. New in the 2026-10-18 PR batch (PR #244); updated to revision 9d1962f in PR #260. Public 0.3 run from PR #260 adopted (kit rescore 75.34, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving at revision 8f9f8619, whose weights are identical to 9d1962f; decision agreement with the submitted rows 99.85% on 831 rows (bar 98.5%). Weights are public on the Hugging Face Hub under a proprietary license. Training data not disclosed. One private new-domain test is not counted for this model: our checks found signs its training data overlapped it. Its new-domain score uses the remaining tests, put on the same scale with the same stock models. Decision Index 0.4.1 (release-v4), source generated 2026-10-29T14:04:16Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Source notes: New in the 2026-10-18 PR batch (PR #244); updated to revision 9d1962f in PR #260. Public 0.3 run from PR #260 adopted (kit rescore 75.34, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving at revision 8f9f8619, whose weights are identical to 9d1962f; decision agreement with the submitted rows 99.85% on 831 rows (bar 98.5%). Weights are public on the Hugging Face Hub under a proprietary license. Training data not disclosed. One private new-domain test is not counted for this model: our checks found signs its training data overlapped it. Its new-domain score uses the remaining tests, put on the same scale with the same stock models. Author public-run adoption/execution status and private lab setup are disclosed in configuration; evaluator-published does not mean every public run was independently executed. 32 benchmarks with a per-field right or wrong answer (ToolRet, BRIGHT and RouterBench excluded), each weighted equally; a 1-in-6 sample of requests; confidence is the probability placed on the chosen option. Rows an entrant trained on count as wrong here too. Contributing benchmarks 31. The board served 9d1962f193; the registry pins the newest repository revision, and the commit titles after it concern the card. Decision Index 0.4 (release-v4), source generated 2026-10-20T14:13:56Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Checked / reported: 2026-11-02 · Samples: 34411 Snapshot: 2026-10-20T14:13:56Z · Revision: 133313cf8963f2b6bed960abb0152fd14cbb6731; c130faf65f82e81d3bdefb01e758c712dc5c5199 for the 2 November 2026 additions · Published observations; result reuse terms not specified Result source · Benchmark source
HF text calibration 0.4 · Brier	0.2377 Brier	Setup & source HF text calibration 0.4 · Brier Exact source engine mammoth-1; {"kind": "full fine-tune", "technique": "autoregressive", "technique_name": "autoregressive", "served_params": 27356728560, "params_estimated": false, "params_from_name": false, "trained_bytes": null, "model_url": "https://huggingface.co/baselabs/mammoth-1", "code_url": null, "base_model": "Qwen/Qwen3.5-27B", "base_url": "https://huggingface.co/Qwen/Qwen3.5-27B", "weights_repo": "baselabs/mammoth-1", "served_checkpoint": "9d1962f193e48a68cf0f3598dc7fb4462fd23ad9", "org": {"id": "baselabs", "name": "baselabs", "logo": "data/orgs/baselabs.png"}, "pr": 260, "new": true, "show_notes": true}. New in the 2026-10-18 PR batch (PR #244); updated to revision 9d1962f in PR #260. Public 0.3 run from PR #260 adopted (kit rescore 75.34, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving at revision 8f9f8619, whose weights are identical to 9d1962f; decision agreement with the submitted rows 99.85% on 831 rows (bar 98.5%). Weights are public on the Hugging Face Hub under a proprietary license. Training data not disclosed. One private new-domain test is not counted for this model: our checks found signs its training data overlapped it. Its new-domain score uses the remaining tests, put on the same scale with the same stock models. Decision Index 0.4.1 (release-v4), source generated 2026-10-29T14:04:16Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Source notes: New in the 2026-10-18 PR batch (PR #244); updated to revision 9d1962f in PR #260. Public 0.3 run from PR #260 adopted (kit rescore 75.34, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving at revision 8f9f8619, whose weights are identical to 9d1962f; decision agreement with the submitted rows 99.85% on 831 rows (bar 98.5%). Weights are public on the Hugging Face Hub under a proprietary license. Training data not disclosed. One private new-domain test is not counted for this model: our checks found signs its training data overlapped it. Its new-domain score uses the remaining tests, put on the same scale with the same stock models. Author public-run adoption/execution status and private lab setup are disclosed in configuration; evaluator-published does not mean every public run was independently executed. 32 benchmarks with a per-field right or wrong answer (ToolRet, BRIGHT and RouterBench excluded), each weighted equally; a 1-in-6 sample of requests; confidence is the probability placed on the chosen option. Rows an entrant trained on count as wrong here too. Contributing benchmarks 31. The board served 9d1962f193; the registry pins the newest repository revision, and the commit titles after it concern the card. Decision Index 0.4 (release-v4), source generated 2026-10-20T14:13:56Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Checked / reported: 2026-11-02 · Samples: 34411 Snapshot: 2026-10-20T14:13:56Z · Revision: 133313cf8963f2b6bed960abb0152fd14cbb6731; c130faf65f82e81d3bdefb01e758c712dc5c5199 for the 2 November 2026 additions · Published observations; result reuse terms not specified Result source · Benchmark source
HF text calibration 0.4 · Sample accuracy	82.66%	Setup & source HF text calibration 0.4 · Sample accuracy Exact source engine mammoth-1; {"kind": "full fine-tune", "technique": "autoregressive", "technique_name": "autoregressive", "served_params": 27356728560, "params_estimated": false, "params_from_name": false, "trained_bytes": null, "model_url": "https://huggingface.co/baselabs/mammoth-1", "code_url": null, "base_model": "Qwen/Qwen3.5-27B", "base_url": "https://huggingface.co/Qwen/Qwen3.5-27B", "weights_repo": "baselabs/mammoth-1", "served_checkpoint": "9d1962f193e48a68cf0f3598dc7fb4462fd23ad9", "org": {"id": "baselabs", "name": "baselabs", "logo": "data/orgs/baselabs.png"}, "pr": 260, "new": true, "show_notes": true}. New in the 2026-10-18 PR batch (PR #244); updated to revision 9d1962f in PR #260. Public 0.3 run from PR #260 adopted (kit rescore 75.34, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving at revision 8f9f8619, whose weights are identical to 9d1962f; decision agreement with the submitted rows 99.85% on 831 rows (bar 98.5%). Weights are public on the Hugging Face Hub under a proprietary license. Training data not disclosed. One private new-domain test is not counted for this model: our checks found signs its training data overlapped it. Its new-domain score uses the remaining tests, put on the same scale with the same stock models. Decision Index 0.4.1 (release-v4), source generated 2026-10-29T14:04:16Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Source notes: New in the 2026-10-18 PR batch (PR #244); updated to revision 9d1962f in PR #260. Public 0.3 run from PR #260 adopted (kit rescore 75.34, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving at revision 8f9f8619, whose weights are identical to 9d1962f; decision agreement with the submitted rows 99.85% on 831 rows (bar 98.5%). Weights are public on the Hugging Face Hub under a proprietary license. Training data not disclosed. One private new-domain test is not counted for this model: our checks found signs its training data overlapped it. Its new-domain score uses the remaining tests, put on the same scale with the same stock models. Author public-run adoption/execution status and private lab setup are disclosed in configuration; evaluator-published does not mean every public run was independently executed. 32 benchmarks with a per-field right or wrong answer (ToolRet, BRIGHT and RouterBench excluded), each weighted equally; a 1-in-6 sample of requests; confidence is the probability placed on the chosen option. Rows an entrant trained on count as wrong here too. Contributing benchmarks 31. The board served 9d1962f193; the registry pins the newest repository revision, and the commit titles after it concern the card. Decision Index 0.4 (release-v4), source generated 2026-10-20T14:13:56Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Checked / reported: 2026-11-02 · Samples: 34411 Snapshot: 2026-10-20T14:13:56Z · Revision: 133313cf8963f2b6bed960abb0152fd14cbb6731; c130faf65f82e81d3bdefb01e758c712dc5c5199 for the 2 November 2026 additions · Published observations; result reuse terms not specified Result source · Benchmark source
HF text calibration 0.4 · Mean chosen confidence	86.53%	Setup & source HF text calibration 0.4 · Mean chosen confidence Exact source engine mammoth-1; {"kind": "full fine-tune", "technique": "autoregressive", "technique_name": "autoregressive", "served_params": 27356728560, "params_estimated": false, "params_from_name": false, "trained_bytes": null, "model_url": "https://huggingface.co/baselabs/mammoth-1", "code_url": null, "base_model": "Qwen/Qwen3.5-27B", "base_url": "https://huggingface.co/Qwen/Qwen3.5-27B", "weights_repo": "baselabs/mammoth-1", "served_checkpoint": "9d1962f193e48a68cf0f3598dc7fb4462fd23ad9", "org": {"id": "baselabs", "name": "baselabs", "logo": "data/orgs/baselabs.png"}, "pr": 260, "new": true, "show_notes": true}. New in the 2026-10-18 PR batch (PR #244); updated to revision 9d1962f in PR #260. Public 0.3 run from PR #260 adopted (kit rescore 75.34, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving at revision 8f9f8619, whose weights are identical to 9d1962f; decision agreement with the submitted rows 99.85% on 831 rows (bar 98.5%). Weights are public on the Hugging Face Hub under a proprietary license. Training data not disclosed. One private new-domain test is not counted for this model: our checks found signs its training data overlapped it. Its new-domain score uses the remaining tests, put on the same scale with the same stock models. Decision Index 0.4.1 (release-v4), source generated 2026-10-29T14:04:16Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Source notes: New in the 2026-10-18 PR batch (PR #244); updated to revision 9d1962f in PR #260. Public 0.3 run from PR #260 adopted (kit rescore 75.34, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving at revision 8f9f8619, whose weights are identical to 9d1962f; decision agreement with the submitted rows 99.85% on 831 rows (bar 98.5%). Weights are public on the Hugging Face Hub under a proprietary license. Training data not disclosed. One private new-domain test is not counted for this model: our checks found signs its training data overlapped it. Its new-domain score uses the remaining tests, put on the same scale with the same stock models. Author public-run adoption/execution status and private lab setup are disclosed in configuration; evaluator-published does not mean every public run was independently executed. 32 benchmarks with a per-field right or wrong answer (ToolRet, BRIGHT and RouterBench excluded), each weighted equally; a 1-in-6 sample of requests; confidence is the probability placed on the chosen option. Rows an entrant trained on count as wrong here too. Contributing benchmarks 31. The board served 9d1962f193; the registry pins the newest repository revision, and the commit titles after it concern the card. Decision Index 0.4 (release-v4), source generated 2026-10-20T14:13:56Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Checked / reported: 2026-11-02 · Samples: 34411 Snapshot: 2026-10-20T14:13:56Z · Revision: 133313cf8963f2b6bed960abb0152fd14cbb6731; c130faf65f82e81d3bdefb01e758c712dc5c5199 for the 2 November 2026 additions · Published observations; result reuse terms not specified Result source · Benchmark source
HF text calibration 0.4 · Wrong among answers at ≥95% confidence	1.37%	Setup & source HF text calibration 0.4 · Wrong among answers at ≥95% confidence Exact source engine mammoth-1; {"kind": "full fine-tune", "technique": "autoregressive", "technique_name": "autoregressive", "served_params": 27356728560, "params_estimated": false, "params_from_name": false, "trained_bytes": null, "model_url": "https://huggingface.co/baselabs/mammoth-1", "code_url": null, "base_model": "Qwen/Qwen3.5-27B", "base_url": "https://huggingface.co/Qwen/Qwen3.5-27B", "weights_repo": "baselabs/mammoth-1", "served_checkpoint": "9d1962f193e48a68cf0f3598dc7fb4462fd23ad9", "org": {"id": "baselabs", "name": "baselabs", "logo": "data/orgs/baselabs.png"}, "pr": 260, "new": true, "show_notes": true}. New in the 2026-10-18 PR batch (PR #244); updated to revision 9d1962f in PR #260. Public 0.3 run from PR #260 adopted (kit rescore 75.34, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving at revision 8f9f8619, whose weights are identical to 9d1962f; decision agreement with the submitted rows 99.85% on 831 rows (bar 98.5%). Weights are public on the Hugging Face Hub under a proprietary license. Training data not disclosed. One private new-domain test is not counted for this model: our checks found signs its training data overlapped it. Its new-domain score uses the remaining tests, put on the same scale with the same stock models. Decision Index 0.4.1 (release-v4), source generated 2026-10-29T14:04:16Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Source notes: New in the 2026-10-18 PR batch (PR #244); updated to revision 9d1962f in PR #260. Public 0.3 run from PR #260 adopted (kit rescore 75.34, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving at revision 8f9f8619, whose weights are identical to 9d1962f; decision agreement with the submitted rows 99.85% on 831 rows (bar 98.5%). Weights are public on the Hugging Face Hub under a proprietary license. Training data not disclosed. One private new-domain test is not counted for this model: our checks found signs its training data overlapped it. Its new-domain score uses the remaining tests, put on the same scale with the same stock models. Author public-run adoption/execution status and private lab setup are disclosed in configuration; evaluator-published does not mean every public run was independently executed. 32 benchmarks with a per-field right or wrong answer (ToolRet, BRIGHT and RouterBench excluded), each weighted equally; a 1-in-6 sample of requests; confidence is the probability placed on the chosen option. Rows an entrant trained on count as wrong here too. Contributing benchmarks 31. The board served 9d1962f193; the registry pins the newest repository revision, and the commit titles after it concern the card. Decision Index 0.4 (release-v4), source generated 2026-10-20T14:13:56Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Checked / reported: 2026-11-02 · Samples: 34411 Snapshot: 2026-10-20T14:13:56Z · Revision: 133313cf8963f2b6bed960abb0152fd14cbb6731; c130faf65f82e81d3bdefb01e758c712dc5c5199 for the 2 November 2026 additions · Published observations; result reuse terms not specified Result source · Benchmark source
HF text calibration 0.4 · Knowledge · ECE	5.63 pp	Setup & source HF text calibration 0.4 · Knowledge · ECE Exact source engine mammoth-1; {"kind": "full fine-tune", "technique": "autoregressive", "technique_name": "autoregressive", "served_params": 27356728560, "params_estimated": false, "params_from_name": false, "trained_bytes": null, "model_url": "https://huggingface.co/baselabs/mammoth-1", "code_url": null, "base_model": "Qwen/Qwen3.5-27B", "base_url": "https://huggingface.co/Qwen/Qwen3.5-27B", "weights_repo": "baselabs/mammoth-1", "served_checkpoint": "9d1962f193e48a68cf0f3598dc7fb4462fd23ad9", "org": {"id": "baselabs", "name": "baselabs", "logo": "data/orgs/baselabs.png"}, "pr": 260, "new": true, "show_notes": true}. New in the 2026-10-18 PR batch (PR #244); updated to revision 9d1962f in PR #260. Public 0.3 run from PR #260 adopted (kit rescore 75.34, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving at revision 8f9f8619, whose weights are identical to 9d1962f; decision agreement with the submitted rows 99.85% on 831 rows (bar 98.5%). Weights are public on the Hugging Face Hub under a proprietary license. Training data not disclosed. One private new-domain test is not counted for this model: our checks found signs its training data overlapped it. Its new-domain score uses the remaining tests, put on the same scale with the same stock models. Decision Index 0.4.1 (release-v4), source generated 2026-10-29T14:04:16Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Source notes: New in the 2026-10-18 PR batch (PR #244); updated to revision 9d1962f in PR #260. Public 0.3 run from PR #260 adopted (kit rescore 75.34, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving at revision 8f9f8619, whose weights are identical to 9d1962f; decision agreement with the submitted rows 99.85% on 831 rows (bar 98.5%). Weights are public on the Hugging Face Hub under a proprietary license. Training data not disclosed. One private new-domain test is not counted for this model: our checks found signs its training data overlapped it. Its new-domain score uses the remaining tests, put on the same scale with the same stock models. Author public-run adoption/execution status and private lab setup are disclosed in configuration; evaluator-published does not mean every public run was independently executed. Area-specific item counts unpublished; do not use whole-sample count. The board served 9d1962f193; the registry pins the newest repository revision, and the commit titles after it concern the card. Decision Index 0.4 (release-v4), source generated 2026-10-20T14:13:56Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Checked / reported: 2026-11-02 · Samples: Not published Snapshot: 2026-10-20T14:13:56Z · Revision: 133313cf8963f2b6bed960abb0152fd14cbb6731; c130faf65f82e81d3bdefb01e758c712dc5c5199 for the 2 November 2026 additions · Published observations; result reuse terms not specified Result source · Benchmark source
HF text calibration 0.4 · Knowledge · Sample accuracy	77.90%	Setup & source HF text calibration 0.4 · Knowledge · Sample accuracy Exact source engine mammoth-1; {"kind": "full fine-tune", "technique": "autoregressive", "technique_name": "autoregressive", "served_params": 27356728560, "params_estimated": false, "params_from_name": false, "trained_bytes": null, "model_url": "https://huggingface.co/baselabs/mammoth-1", "code_url": null, "base_model": "Qwen/Qwen3.5-27B", "base_url": "https://huggingface.co/Qwen/Qwen3.5-27B", "weights_repo": "baselabs/mammoth-1", "served_checkpoint": "9d1962f193e48a68cf0f3598dc7fb4462fd23ad9", "org": {"id": "baselabs", "name": "baselabs", "logo": "data/orgs/baselabs.png"}, "pr": 260, "new": true, "show_notes": true}. New in the 2026-10-18 PR batch (PR #244); updated to revision 9d1962f in PR #260. Public 0.3 run from PR #260 adopted (kit rescore 75.34, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving at revision 8f9f8619, whose weights are identical to 9d1962f; decision agreement with the submitted rows 99.85% on 831 rows (bar 98.5%). Weights are public on the Hugging Face Hub under a proprietary license. Training data not disclosed. One private new-domain test is not counted for this model: our checks found signs its training data overlapped it. Its new-domain score uses the remaining tests, put on the same scale with the same stock models. Decision Index 0.4.1 (release-v4), source generated 2026-10-29T14:04:16Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Source notes: New in the 2026-10-18 PR batch (PR #244); updated to revision 9d1962f in PR #260. Public 0.3 run from PR #260 adopted (kit rescore 75.34, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving at revision 8f9f8619, whose weights are identical to 9d1962f; decision agreement with the submitted rows 99.85% on 831 rows (bar 98.5%). Weights are public on the Hugging Face Hub under a proprietary license. Training data not disclosed. One private new-domain test is not counted for this model: our checks found signs its training data overlapped it. Its new-domain score uses the remaining tests, put on the same scale with the same stock models. Author public-run adoption/execution status and private lab setup are disclosed in configuration; evaluator-published does not mean every public run was independently executed. Area-specific item counts unpublished; do not use whole-sample count. The board served 9d1962f193; the registry pins the newest repository revision, and the commit titles after it concern the card. Decision Index 0.4 (release-v4), source generated 2026-10-20T14:13:56Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Checked / reported: 2026-11-02 · Samples: Not published Snapshot: 2026-10-20T14:13:56Z · Revision: 133313cf8963f2b6bed960abb0152fd14cbb6731; c130faf65f82e81d3bdefb01e758c712dc5c5199 for the 2 November 2026 additions · Published observations; result reuse terms not specified Result source · Benchmark source
HF text calibration 0.4 · Knowledge · Mean chosen confidence	82.85%	Setup & source HF text calibration 0.4 · Knowledge · Mean chosen confidence Exact source engine mammoth-1; {"kind": "full fine-tune", "technique": "autoregressive", "technique_name": "autoregressive", "served_params": 27356728560, "params_estimated": false, "params_from_name": false, "trained_bytes": null, "model_url": "https://huggingface.co/baselabs/mammoth-1", "code_url": null, "base_model": "Qwen/Qwen3.5-27B", "base_url": "https://huggingface.co/Qwen/Qwen3.5-27B", "weights_repo": "baselabs/mammoth-1", "served_checkpoint": "9d1962f193e48a68cf0f3598dc7fb4462fd23ad9", "org": {"id": "baselabs", "name": "baselabs", "logo": "data/orgs/baselabs.png"}, "pr": 260, "new": true, "show_notes": true}. New in the 2026-10-18 PR batch (PR #244); updated to revision 9d1962f in PR #260. Public 0.3 run from PR #260 adopted (kit rescore 75.34, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving at revision 8f9f8619, whose weights are identical to 9d1962f; decision agreement with the submitted rows 99.85% on 831 rows (bar 98.5%). Weights are public on the Hugging Face Hub under a proprietary license. Training data not disclosed. One private new-domain test is not counted for this model: our checks found signs its training data overlapped it. Its new-domain score uses the remaining tests, put on the same scale with the same stock models. Decision Index 0.4.1 (release-v4), source generated 2026-10-29T14:04:16Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Source notes: New in the 2026-10-18 PR batch (PR #244); updated to revision 9d1962f in PR #260. Public 0.3 run from PR #260 adopted (kit rescore 75.34, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving at revision 8f9f8619, whose weights are identical to 9d1962f; decision agreement with the submitted rows 99.85% on 831 rows (bar 98.5%). Weights are public on the Hugging Face Hub under a proprietary license. Training data not disclosed. One private new-domain test is not counted for this model: our checks found signs its training data overlapped it. Its new-domain score uses the remaining tests, put on the same scale with the same stock models. Author public-run adoption/execution status and private lab setup are disclosed in configuration; evaluator-published does not mean every public run was independently executed. Area-specific item counts unpublished; do not use whole-sample count. The board served 9d1962f193; the registry pins the newest repository revision, and the commit titles after it concern the card. Decision Index 0.4 (release-v4), source generated 2026-10-20T14:13:56Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Checked / reported: 2026-11-02 · Samples: Not published Snapshot: 2026-10-20T14:13:56Z · Revision: 133313cf8963f2b6bed960abb0152fd14cbb6731; c130faf65f82e81d3bdefb01e758c712dc5c5199 for the 2 November 2026 additions · Published observations; result reuse terms not specified Result source · Benchmark source
HF text calibration 0.4 · Language · ECE	1.53 pp	Setup & source HF text calibration 0.4 · Language · ECE Exact source engine mammoth-1; {"kind": "full fine-tune", "technique": "autoregressive", "technique_name": "autoregressive", "served_params": 27356728560, "params_estimated": false, "params_from_name": false, "trained_bytes": null, "model_url": "https://huggingface.co/baselabs/mammoth-1", "code_url": null, "base_model": "Qwen/Qwen3.5-27B", "base_url": "https://huggingface.co/Qwen/Qwen3.5-27B", "weights_repo": "baselabs/mammoth-1", "served_checkpoint": "9d1962f193e48a68cf0f3598dc7fb4462fd23ad9", "org": {"id": "baselabs", "name": "baselabs", "logo": "data/orgs/baselabs.png"}, "pr": 260, "new": true, "show_notes": true}. New in the 2026-10-18 PR batch (PR #244); updated to revision 9d1962f in PR #260. Public 0.3 run from PR #260 adopted (kit rescore 75.34, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving at revision 8f9f8619, whose weights are identical to 9d1962f; decision agreement with the submitted rows 99.85% on 831 rows (bar 98.5%). Weights are public on the Hugging Face Hub under a proprietary license. Training data not disclosed. One private new-domain test is not counted for this model: our checks found signs its training data overlapped it. Its new-domain score uses the remaining tests, put on the same scale with the same stock models. Decision Index 0.4.1 (release-v4), source generated 2026-10-29T14:04:16Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Source notes: New in the 2026-10-18 PR batch (PR #244); updated to revision 9d1962f in PR #260. Public 0.3 run from PR #260 adopted (kit rescore 75.34, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving at revision 8f9f8619, whose weights are identical to 9d1962f; decision agreement with the submitted rows 99.85% on 831 rows (bar 98.5%). Weights are public on the Hugging Face Hub under a proprietary license. Training data not disclosed. One private new-domain test is not counted for this model: our checks found signs its training data overlapped it. Its new-domain score uses the remaining tests, put on the same scale with the same stock models. Author public-run adoption/execution status and private lab setup are disclosed in configuration; evaluator-published does not mean every public run was independently executed. Area-specific item counts unpublished; do not use whole-sample count. The board served 9d1962f193; the registry pins the newest repository revision, and the commit titles after it concern the card. Decision Index 0.4 (release-v4), source generated 2026-10-20T14:13:56Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Checked / reported: 2026-11-02 · Samples: Not published Snapshot: 2026-10-20T14:13:56Z · Revision: 133313cf8963f2b6bed960abb0152fd14cbb6731; c130faf65f82e81d3bdefb01e758c712dc5c5199 for the 2 November 2026 additions · Published observations; result reuse terms not specified Result source · Benchmark source
HF text calibration 0.4 · Language · Sample accuracy	91.35%	Setup & source HF text calibration 0.4 · Language · Sample accuracy Exact source engine mammoth-1; {"kind": "full fine-tune", "technique": "autoregressive", "technique_name": "autoregressive", "served_params": 27356728560, "params_estimated": false, "params_from_name": false, "trained_bytes": null, "model_url": "https://huggingface.co/baselabs/mammoth-1", "code_url": null, "base_model": "Qwen/Qwen3.5-27B", "base_url": "https://huggingface.co/Qwen/Qwen3.5-27B", "weights_repo": "baselabs/mammoth-1", "served_checkpoint": "9d1962f193e48a68cf0f3598dc7fb4462fd23ad9", "org": {"id": "baselabs", "name": "baselabs", "logo": "data/orgs/baselabs.png"}, "pr": 260, "new": true, "show_notes": true}. New in the 2026-10-18 PR batch (PR #244); updated to revision 9d1962f in PR #260. Public 0.3 run from PR #260 adopted (kit rescore 75.34, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving at revision 8f9f8619, whose weights are identical to 9d1962f; decision agreement with the submitted rows 99.85% on 831 rows (bar 98.5%). Weights are public on the Hugging Face Hub under a proprietary license. Training data not disclosed. One private new-domain test is not counted for this model: our checks found signs its training data overlapped it. Its new-domain score uses the remaining tests, put on the same scale with the same stock models. Decision Index 0.4.1 (release-v4), source generated 2026-10-29T14:04:16Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Source notes: New in the 2026-10-18 PR batch (PR #244); updated to revision 9d1962f in PR #260. Public 0.3 run from PR #260 adopted (kit rescore 75.34, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving at revision 8f9f8619, whose weights are identical to 9d1962f; decision agreement with the submitted rows 99.85% on 831 rows (bar 98.5%). Weights are public on the Hugging Face Hub under a proprietary license. Training data not disclosed. One private new-domain test is not counted for this model: our checks found signs its training data overlapped it. Its new-domain score uses the remaining tests, put on the same scale with the same stock models. Author public-run adoption/execution status and private lab setup are disclosed in configuration; evaluator-published does not mean every public run was independently executed. Area-specific item counts unpublished; do not use whole-sample count. The board served 9d1962f193; the registry pins the newest repository revision, and the commit titles after it concern the card. Decision Index 0.4 (release-v4), source generated 2026-10-20T14:13:56Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Checked / reported: 2026-11-02 · Samples: Not published Snapshot: 2026-10-20T14:13:56Z · Revision: 133313cf8963f2b6bed960abb0152fd14cbb6731; c130faf65f82e81d3bdefb01e758c712dc5c5199 for the 2 November 2026 additions · Published observations; result reuse terms not specified Result source · Benchmark source
HF text calibration 0.4 · Language · Mean chosen confidence	92.71%	Setup & source HF text calibration 0.4 · Language · Mean chosen confidence Exact source engine mammoth-1; {"kind": "full fine-tune", "technique": "autoregressive", "technique_name": "autoregressive", "served_params": 27356728560, "params_estimated": false, "params_from_name": false, "trained_bytes": null, "model_url": "https://huggingface.co/baselabs/mammoth-1", "code_url": null, "base_model": "Qwen/Qwen3.5-27B", "base_url": "https://huggingface.co/Qwen/Qwen3.5-27B", "weights_repo": "baselabs/mammoth-1", "served_checkpoint": "9d1962f193e48a68cf0f3598dc7fb4462fd23ad9", "org": {"id": "baselabs", "name": "baselabs", "logo": "data/orgs/baselabs.png"}, "pr": 260, "new": true, "show_notes": true}. New in the 2026-10-18 PR batch (PR #244); updated to revision 9d1962f in PR #260. Public 0.3 run from PR #260 adopted (kit rescore 75.34, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving at revision 8f9f8619, whose weights are identical to 9d1962f; decision agreement with the submitted rows 99.85% on 831 rows (bar 98.5%). Weights are public on the Hugging Face Hub under a proprietary license. Training data not disclosed. One private new-domain test is not counted for this model: our checks found signs its training data overlapped it. Its new-domain score uses the remaining tests, put on the same scale with the same stock models. Decision Index 0.4.1 (release-v4), source generated 2026-10-29T14:04:16Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Source notes: New in the 2026-10-18 PR batch (PR #244); updated to revision 9d1962f in PR #260. Public 0.3 run from PR #260 adopted (kit rescore 75.34, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving at revision 8f9f8619, whose weights are identical to 9d1962f; decision agreement with the submitted rows 99.85% on 831 rows (bar 98.5%). Weights are public on the Hugging Face Hub under a proprietary license. Training data not disclosed. One private new-domain test is not counted for this model: our checks found signs its training data overlapped it. Its new-domain score uses the remaining tests, put on the same scale with the same stock models. Author public-run adoption/execution status and private lab setup are disclosed in configuration; evaluator-published does not mean every public run was independently executed. Area-specific item counts unpublished; do not use whole-sample count. The board served 9d1962f193; the registry pins the newest repository revision, and the commit titles after it concern the card. Decision Index 0.4 (release-v4), source generated 2026-10-20T14:13:56Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Checked / reported: 2026-11-02 · Samples: Not published Snapshot: 2026-10-20T14:13:56Z · Revision: 133313cf8963f2b6bed960abb0152fd14cbb6731; c130faf65f82e81d3bdefb01e758c712dc5c5199 for the 2 November 2026 additions · Published observations; result reuse terms not specified Result source · Benchmark source
HF text calibration 0.4 · Retrieval · ECE	11.39 pp	Setup & source HF text calibration 0.4 · Retrieval · ECE Exact source engine mammoth-1; {"kind": "full fine-tune", "technique": "autoregressive", "technique_name": "autoregressive", "served_params": 27356728560, "params_estimated": false, "params_from_name": false, "trained_bytes": null, "model_url": "https://huggingface.co/baselabs/mammoth-1", "code_url": null, "base_model": "Qwen/Qwen3.5-27B", "base_url": "https://huggingface.co/Qwen/Qwen3.5-27B", "weights_repo": "baselabs/mammoth-1", "served_checkpoint": "9d1962f193e48a68cf0f3598dc7fb4462fd23ad9", "org": {"id": "baselabs", "name": "baselabs", "logo": "data/orgs/baselabs.png"}, "pr": 260, "new": true, "show_notes": true}. New in the 2026-10-18 PR batch (PR #244); updated to revision 9d1962f in PR #260. Public 0.3 run from PR #260 adopted (kit rescore 75.34, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving at revision 8f9f8619, whose weights are identical to 9d1962f; decision agreement with the submitted rows 99.85% on 831 rows (bar 98.5%). Weights are public on the Hugging Face Hub under a proprietary license. Training data not disclosed. One private new-domain test is not counted for this model: our checks found signs its training data overlapped it. Its new-domain score uses the remaining tests, put on the same scale with the same stock models. Decision Index 0.4.1 (release-v4), source generated 2026-10-29T14:04:16Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Source notes: New in the 2026-10-18 PR batch (PR #244); updated to revision 9d1962f in PR #260. Public 0.3 run from PR #260 adopted (kit rescore 75.34, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving at revision 8f9f8619, whose weights are identical to 9d1962f; decision agreement with the submitted rows 99.85% on 831 rows (bar 98.5%). Weights are public on the Hugging Face Hub under a proprietary license. Training data not disclosed. One private new-domain test is not counted for this model: our checks found signs its training data overlapped it. Its new-domain score uses the remaining tests, put on the same scale with the same stock models. Author public-run adoption/execution status and private lab setup are disclosed in configuration; evaluator-published does not mean every public run was independently executed. Area-specific item counts unpublished; do not use whole-sample count. The board served 9d1962f193; the registry pins the newest repository revision, and the commit titles after it concern the card. Decision Index 0.4 (release-v4), source generated 2026-10-20T14:13:56Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Checked / reported: 2026-11-02 · Samples: Not published Snapshot: 2026-10-20T14:13:56Z · Revision: 133313cf8963f2b6bed960abb0152fd14cbb6731; c130faf65f82e81d3bdefb01e758c712dc5c5199 for the 2 November 2026 additions · Published observations; result reuse terms not specified Result source · Benchmark source
HF text calibration 0.4 · Retrieval · Sample accuracy	74.02%	Setup & source HF text calibration 0.4 · Retrieval · Sample accuracy Exact source engine mammoth-1; {"kind": "full fine-tune", "technique": "autoregressive", "technique_name": "autoregressive", "served_params": 27356728560, "params_estimated": false, "params_from_name": false, "trained_bytes": null, "model_url": "https://huggingface.co/baselabs/mammoth-1", "code_url": null, "base_model": "Qwen/Qwen3.5-27B", "base_url": "https://huggingface.co/Qwen/Qwen3.5-27B", "weights_repo": "baselabs/mammoth-1", "served_checkpoint": "9d1962f193e48a68cf0f3598dc7fb4462fd23ad9", "org": {"id": "baselabs", "name": "baselabs", "logo": "data/orgs/baselabs.png"}, "pr": 260, "new": true, "show_notes": true}. New in the 2026-10-18 PR batch (PR #244); updated to revision 9d1962f in PR #260. Public 0.3 run from PR #260 adopted (kit rescore 75.34, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving at revision 8f9f8619, whose weights are identical to 9d1962f; decision agreement with the submitted rows 99.85% on 831 rows (bar 98.5%). Weights are public on the Hugging Face Hub under a proprietary license. Training data not disclosed. One private new-domain test is not counted for this model: our checks found signs its training data overlapped it. Its new-domain score uses the remaining tests, put on the same scale with the same stock models. Decision Index 0.4.1 (release-v4), source generated 2026-10-29T14:04:16Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Source notes: New in the 2026-10-18 PR batch (PR #244); updated to revision 9d1962f in PR #260. Public 0.3 run from PR #260 adopted (kit rescore 75.34, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving at revision 8f9f8619, whose weights are identical to 9d1962f; decision agreement with the submitted rows 99.85% on 831 rows (bar 98.5%). Weights are public on the Hugging Face Hub under a proprietary license. Training data not disclosed. One private new-domain test is not counted for this model: our checks found signs its training data overlapped it. Its new-domain score uses the remaining tests, put on the same scale with the same stock models. Author public-run adoption/execution status and private lab setup are disclosed in configuration; evaluator-published does not mean every public run was independently executed. Area-specific item counts unpublished; do not use whole-sample count. The board served 9d1962f193; the registry pins the newest repository revision, and the commit titles after it concern the card. Decision Index 0.4 (release-v4), source generated 2026-10-20T14:13:56Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Checked / reported: 2026-11-02 · Samples: Not published Snapshot: 2026-10-20T14:13:56Z · Revision: 133313cf8963f2b6bed960abb0152fd14cbb6731; c130faf65f82e81d3bdefb01e758c712dc5c5199 for the 2 November 2026 additions · Published observations; result reuse terms not specified Result source · Benchmark source
HF text calibration 0.4 · Retrieval · Mean chosen confidence	85.41%	Setup & source HF text calibration 0.4 · Retrieval · Mean chosen confidence Exact source engine mammoth-1; {"kind": "full fine-tune", "technique": "autoregressive", "technique_name": "autoregressive", "served_params": 27356728560, "params_estimated": false, "params_from_name": false, "trained_bytes": null, "model_url": "https://huggingface.co/baselabs/mammoth-1", "code_url": null, "base_model": "Qwen/Qwen3.5-27B", "base_url": "https://huggingface.co/Qwen/Qwen3.5-27B", "weights_repo": "baselabs/mammoth-1", "served_checkpoint": "9d1962f193e48a68cf0f3598dc7fb4462fd23ad9", "org": {"id": "baselabs", "name": "baselabs", "logo": "data/orgs/baselabs.png"}, "pr": 260, "new": true, "show_notes": true}. New in the 2026-10-18 PR batch (PR #244); updated to revision 9d1962f in PR #260. Public 0.3 run from PR #260 adopted (kit rescore 75.34, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving at revision 8f9f8619, whose weights are identical to 9d1962f; decision agreement with the submitted rows 99.85% on 831 rows (bar 98.5%). Weights are public on the Hugging Face Hub under a proprietary license. Training data not disclosed. One private new-domain test is not counted for this model: our checks found signs its training data overlapped it. Its new-domain score uses the remaining tests, put on the same scale with the same stock models. Decision Index 0.4.1 (release-v4), source generated 2026-10-29T14:04:16Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Source notes: New in the 2026-10-18 PR batch (PR #244); updated to revision 9d1962f in PR #260. Public 0.3 run from PR #260 adopted (kit rescore 75.34, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving at revision 8f9f8619, whose weights are identical to 9d1962f; decision agreement with the submitted rows 99.85% on 831 rows (bar 98.5%). Weights are public on the Hugging Face Hub under a proprietary license. Training data not disclosed. One private new-domain test is not counted for this model: our checks found signs its training data overlapped it. Its new-domain score uses the remaining tests, put on the same scale with the same stock models. Author public-run adoption/execution status and private lab setup are disclosed in configuration; evaluator-published does not mean every public run was independently executed. Area-specific item counts unpublished; do not use whole-sample count. The board served 9d1962f193; the registry pins the newest repository revision, and the commit titles after it concern the card. Decision Index 0.4 (release-v4), source generated 2026-10-20T14:13:56Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Checked / reported: 2026-11-02 · Samples: Not published Snapshot: 2026-10-20T14:13:56Z · Revision: 133313cf8963f2b6bed960abb0152fd14cbb6731; c130faf65f82e81d3bdefb01e758c712dc5c5199 for the 2 November 2026 additions · Published observations; result reuse terms not specified Result source · Benchmark source