Span-02 Decision Model Results

Span-02

Explore Span-02: exact decision configuration, published specifications, source-linked benchmarks and API tariffs when available.

Published specifications

Provider: Respan
Version: respanai/span-02 (anytime reasoning)
System type: Trained checkpoint
Access: Open weights
License: Apache-2.0; Gemma 4 license link
Inputs: text, image
Decision types: Choice, Yes / no, Score, Number
Published size: 26B
Active parameters: 4B
Context / input limit: 131,072 tokens per question; up to 4 images
Announced: 2026-10-15

Hugging Face · Announcement · Website · OpenRouter

Model notes

Fine-tune of google/gemma-4-26B-A4B-it served by Respan's own engine, with a budget_ms setting that spends more time reasoning. NVFP4/FP8 weights of 16.4 GB; Hugging Face's stored count of 15,815,134,090 is not the 26B model size. lihe07/span-02 redirects to this repository with identical metadata, so board rows naming lihe07 are this checkpoint. OpenRouter endpoint label respan/span-02-20261015. Span-01 is older and not registered. Typed outputs are probabilities over supplied options; confidence is not a correctness guarantee.

Checkpoint revision: 63cbdd774391f1b7d9bc3829179efe996d0191ce

← All decision models · Compare this decision system →

Published API tariff

$0.01 per 1M input tokens

Respan · checked 2026-11-02

Launch post: $0.01 per million input tokens on the Respan Gateway, output free. OpenRouter lists the same rate.

No output-token charge does not mean free requests. These are provider tariffs, separate from benchmark workload costs.

607 recorded observations. Published results retain their source and setup; no Examenos evaluation was run. Missing results are not zero. How to read these comparisons →

Official / vendor · Respan launch post (vendor measured)

Span-02 · Official / vendor · Respan launch post (vendor measured)
Evaluation	Result	Evidence
Respan launch · Decision Index 0.3 public · thinking off	61.2	Setup & source Decision Index 0.3 at latency budget thinking off, read from the figure label. Span-02, thinking off, from the alt text of the launch figure. Printed in the vendor's own table. Respan's own runs. Span-02's budget_ms setting spends time reasoning, so each setting is its own evaluation. Perplexity Decider in the same figure has no named version and Jev 1.13.0 is unregistered, so neither has a row. Not ranked here. Checked / reported: 2026-11-02 · Samples: Not published Snapshot: 2026-10-15 · Revision: Read 2026-11-02 · Published observations; result reuse terms not specified Result source · Benchmark source
Respan launch · Decision Index 0.3 public · 270 ms	61.5	Setup & source Decision Index 0.3 at latency budget 270 ms, read from the figure label. Span-02, 270 ms, from the alt text of the launch figure. Printed in the vendor's own table. Respan's own runs. Span-02's budget_ms setting spends time reasoning, so each setting is its own evaluation. Perplexity Decider in the same figure has no named version and Jev 1.13.0 is unregistered, so neither has a row. Not ranked here. Checked / reported: 2026-11-02 · Samples: Not published Snapshot: 2026-10-15 · Revision: Read 2026-11-02 · Published observations; result reuse terms not specified Result source · Benchmark source
Respan launch · Decision Index 0.3 public · 500 ms	62.5	Setup & source Decision Index 0.3 at latency budget 500 ms, read from the figure label. Span-02, 500 ms, from the alt text of the launch figure. Printed in the vendor's own table. Respan's own runs. Span-02's budget_ms setting spends time reasoning, so each setting is its own evaluation. Perplexity Decider in the same figure has no named version and Jev 1.13.0 is unregistered, so neither has a row. Not ranked here. Checked / reported: 2026-11-02 · Samples: Not published Snapshot: 2026-10-15 · Revision: Read 2026-11-02 · Published observations; result reuse terms not specified Result source · Benchmark source
Respan launch · Decision Index 0.3 public · 1 s	63.0	Setup & source Decision Index 0.3 at latency budget 1 s, read from the figure label. Span-02, 1 s, from the alt text of the launch figure. Printed in the vendor's own table. Respan's own runs. Span-02's budget_ms setting spends time reasoning, so each setting is its own evaluation. Perplexity Decider in the same figure has no named version and Jev 1.13.0 is unregistered, so neither has a row. Not ranked here. Checked / reported: 2026-11-02 · Samples: Not published Snapshot: 2026-10-15 · Revision: Read 2026-11-02 · Published observations; result reuse terms not specified Result source · Benchmark source
Respan · Blind Earth · thinking off	71.9%	Setup & source Is this point in water, from latitude and longitude alone, on a 4° grid of 4,050 points scored against Natural Earth. Span-02, thinking off, one request per point. Printed in the vendor's own table. Respan's own runs. Span-02's budget_ms setting spends time reasoning, so each setting is its own evaluation. Perplexity Decider in the same figure has no named version and Jev 1.13.0 is unregistered, so neither has a row. Not ranked here. Checked / reported: 2026-11-02 · Samples: 4050 Snapshot: 2026-10-15 · Revision: Read 2026-11-02 · Published observations; result reuse terms not specified Result source · Benchmark source
Respan · Blind Earth · 100 ms	72.5%	Setup & source Is this point in water, from latitude and longitude alone, on a 4° grid of 4,050 points scored against Natural Earth. Span-02, 100 ms, one request per point. Printed in the vendor's own table. Respan's own runs. Span-02's budget_ms setting spends time reasoning, so each setting is its own evaluation. Perplexity Decider in the same figure has no named version and Jev 1.13.0 is unregistered, so neither has a row. Not ranked here. Checked / reported: 2026-11-02 · Samples: 4050 Snapshot: 2026-10-15 · Revision: Read 2026-11-02 · Published observations; result reuse terms not specified Result source · Benchmark source
Respan · Blind Earth · 300 ms	84.5%	Setup & source Is this point in water, from latitude and longitude alone, on a 4° grid of 4,050 points scored against Natural Earth. Span-02, 300 ms, one request per point. Printed in the vendor's own table. Respan's own runs. Span-02's budget_ms setting spends time reasoning, so each setting is its own evaluation. Perplexity Decider in the same figure has no named version and Jev 1.13.0 is unregistered, so neither has a row. Not ranked here. Checked / reported: 2026-11-02 · Samples: 4050 Snapshot: 2026-10-15 · Revision: Read 2026-11-02 · Published observations; result reuse terms not specified Result source · Benchmark source
Respan · Blind Earth · 1 s	84.7%	Setup & source Is this point in water, from latitude and longitude alone, on a 4° grid of 4,050 points scored against Natural Earth. Span-02, 1 s, one request per point. Printed in the vendor's own table. Respan's own runs. Span-02's budget_ms setting spends time reasoning, so each setting is its own evaluation. Perplexity Decider in the same figure has no named version and Jev 1.13.0 is unregistered, so neither has a row. Not ranked here. Checked / reported: 2026-11-02 · Samples: 4050 Snapshot: 2026-10-15 · Revision: Read 2026-11-02 · Published observations; result reuse terms not specified Result source · Benchmark source

Official / vendor · Respan Span-02 model card (vendor reported)

Span-02 · Official / vendor · Respan Span-02 model card (vendor reported)
Evaluation	Result	Evidence
Span-02 card · Decision Index Full score	62.6	Setup & source Full score quoted on the card, rank =4 tied with four others when written. Official Decision Index result as quoted on the card. Printed in the vendor's own table. The card's quoted Decision Index result, from the board when the card was written. Current community 0.4.1 rows are stored separately. Not ranked here. Checked / reported: 2026-11-02 · Samples: 110201 Snapshot: 2026-10-15 · Revision: 63cbdd774391f1b7d9bc3829179efe996d0191ce · Published observations; result reuse terms not specified Result source · Benchmark source
Span-02 card · Median latency	33 ms	Setup & source Median across the 110,201 answered requests. Official Decision Index result as quoted on the card. Printed in the vendor's own table. The card's quoted Decision Index result, from the board when the card was written. Current community 0.4.1 rows are stored separately. Not ranked here. Checked / reported: 2026-11-02 · Samples: 110201 Snapshot: 2026-10-15 · Revision: 63cbdd774391f1b7d9bc3829179efe996d0191ce · Published observations; result reuse terms not specified Result source · Benchmark source
Span-02 card · P95 latency	822 ms	Setup & source P95 across the 110,201 answered requests. Official Decision Index result as quoted on the card. Printed in the vendor's own table. The card's quoted Decision Index result, from the board when the card was written. Current community 0.4.1 rows are stored separately. Not ranked here. Checked / reported: 2026-11-02 · Samples: 110201 Snapshot: 2026-10-15 · Revision: 63cbdd774391f1b7d9bc3829179efe996d0191ce · Published observations; result reuse terms not specified Result source · Benchmark source

Evaluator-published · Community Decision Index

Evaluator-published · etsabary System 1 Decision Benchmark

Span-02 · Evaluator-published · etsabary System 1 Decision Benchmark
Evaluation	Result	Evidence
System 1 Decision Benchmark v2 · All-question accuracy	73.681%	Setup & source Correct answers over all 1,934 questions. Respan Span-02 respan/span-02-20261015; Hosted native; returned identities {"respan/span-02-20261015": 1934}; collected 2026-10-18 to 2026-10-31 America/Toronto; one primary response per input, no scored retries. Exact aggregate from the release statistics. Development benchmark, not an independently sampled test: 1,934 questions in 41 categories, the frozen 1,342-question bank plus 192 logical and 400 practical additions. The bank was built with model-guided development, and Jev plus retained Microsoft and Drex feedback are development exposure. Related variants and a single recorded primary response per input limit generalization. The scoring key stays private, so public checks cover coverage, timing and aggregate consistency but cannot rescore answers. Strict accuracy counts a wrong usable answer and an unsuccessful response in the same fixed denominator. Timing pools primary client times with the network included; providers, precision, concurrency, routes and dates differ, retained and new timings can span sessions, and quick failures can reduce all-primary latency. The source published a separate cost ledger on 1 November 2026, stored as decision-reference-v2-cost-per-1000, which supersedes its earlier “cost not consolidated” note. Older 1,342-only scores are not converted into this bank. Not ranked here. Checked / reported: 2026-11-02 · Samples: 1934 Snapshot: 2026-11-01 · Revision: f5fa9cfafccd97fcfbdb0336e14284faaa8d2d35 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
System 1 Decision Benchmark v2 · Macro-category accuracy	72.211%	Setup & source Equal-weight mean over the 41 categories. Respan Span-02 respan/span-02-20261015; Hosted native; returned identities {"respan/span-02-20261015": 1934}; collected 2026-10-18 to 2026-10-31 America/Toronto; one primary response per input, no scored retries. Exact aggregate from the release statistics. Development benchmark, not an independently sampled test: 1,934 questions in 41 categories, the frozen 1,342-question bank plus 192 logical and 400 practical additions. The bank was built with model-guided development, and Jev plus retained Microsoft and Drex feedback are development exposure. Related variants and a single recorded primary response per input limit generalization. The scoring key stays private, so public checks cover coverage, timing and aggregate consistency but cannot rescore answers. Strict accuracy counts a wrong usable answer and an unsuccessful response in the same fixed denominator. Timing pools primary client times with the network included; providers, precision, concurrency, routes and dates differ, retained and new timings can span sessions, and quick failures can reduce all-primary latency. The source published a separate cost ledger on 1 November 2026, stored as decision-reference-v2-cost-per-1000, which supersedes its earlier “cost not consolidated” note. Older 1,342-only scores are not converted into this bank. Not ranked here. Checked / reported: 2026-11-02 · Samples: 1934 Snapshot: 2026-11-01 · Revision: f5fa9cfafccd97fcfbdb0336e14284faaa8d2d35 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
System 1 Decision Benchmark v2 · Added 592 questions	50.338%	Setup & source Accuracy on the 592 questions new in v2. Respan Span-02 respan/span-02-20261015; Hosted native; returned identities {"respan/span-02-20261015": 1934}; collected 2026-10-18 to 2026-10-31 America/Toronto; one primary response per input, no scored retries. Exact aggregate from the release statistics. Development benchmark, not an independently sampled test: 1,934 questions in 41 categories, the frozen 1,342-question bank plus 192 logical and 400 practical additions. The bank was built with model-guided development, and Jev plus retained Microsoft and Drex feedback are development exposure. Related variants and a single recorded primary response per input limit generalization. The scoring key stays private, so public checks cover coverage, timing and aggregate consistency but cannot rescore answers. Strict accuracy counts a wrong usable answer and an unsuccessful response in the same fixed denominator. Timing pools primary client times with the network included; providers, precision, concurrency, routes and dates differ, retained and new timings can span sessions, and quick failures can reduce all-primary latency. The source published a separate cost ledger on 1 November 2026, stored as decision-reference-v2-cost-per-1000, which supersedes its earlier “cost not consolidated” note. Older 1,342-only scores are not converted into this bank. Not ranked here. Checked / reported: 2026-11-02 · Samples: 592 Snapshot: 2026-11-01 · Revision: f5fa9cfafccd97fcfbdb0336e14284faaa8d2d35 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
System 1 Decision Benchmark v2 · Logical additions	49.479%	Setup & source Accuracy on the 192 logical additions. Respan Span-02 respan/span-02-20261015; Hosted native; returned identities {"respan/span-02-20261015": 1934}; collected 2026-10-18 to 2026-10-31 America/Toronto; one primary response per input, no scored retries. Exact aggregate from the release statistics. Development benchmark, not an independently sampled test: 1,934 questions in 41 categories, the frozen 1,342-question bank plus 192 logical and 400 practical additions. The bank was built with model-guided development, and Jev plus retained Microsoft and Drex feedback are development exposure. Related variants and a single recorded primary response per input limit generalization. The scoring key stays private, so public checks cover coverage, timing and aggregate consistency but cannot rescore answers. Strict accuracy counts a wrong usable answer and an unsuccessful response in the same fixed denominator. Timing pools primary client times with the network included; providers, precision, concurrency, routes and dates differ, retained and new timings can span sessions, and quick failures can reduce all-primary latency. The source published a separate cost ledger on 1 November 2026, stored as decision-reference-v2-cost-per-1000, which supersedes its earlier “cost not consolidated” note. Older 1,342-only scores are not converted into this bank. Not ranked here. Checked / reported: 2026-11-02 · Samples: 192 Snapshot: 2026-11-01 · Revision: f5fa9cfafccd97fcfbdb0336e14284faaa8d2d35 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
System 1 Decision Benchmark v2 · Practical additions	50.750%	Setup & source Accuracy on the 400 practical additions. Respan Span-02 respan/span-02-20261015; Hosted native; returned identities {"respan/span-02-20261015": 1934}; collected 2026-10-18 to 2026-10-31 America/Toronto; one primary response per input, no scored retries. Exact aggregate from the release statistics. Development benchmark, not an independently sampled test: 1,934 questions in 41 categories, the frozen 1,342-question bank plus 192 logical and 400 practical additions. The bank was built with model-guided development, and Jev plus retained Microsoft and Drex feedback are development exposure. Related variants and a single recorded primary response per input limit generalization. The scoring key stays private, so public checks cover coverage, timing and aggregate consistency but cannot rescore answers. Strict accuracy counts a wrong usable answer and an unsuccessful response in the same fixed denominator. Timing pools primary client times with the network included; providers, precision, concurrency, routes and dates differ, retained and new timings can span sessions, and quick failures can reduce all-primary latency. The source published a separate cost ledger on 1 November 2026, stored as decision-reference-v2-cost-per-1000, which supersedes its earlier “cost not consolidated” note. Older 1,342-only scores are not converted into this bank. Not ranked here. Checked / reported: 2026-11-02 · Samples: 400 Snapshot: 2026-11-01 · Revision: f5fa9cfafccd97fcfbdb0336e14284faaa8d2d35 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
System 1 Decision Benchmark v2 · Original 1,342 questions	83.979%	Setup & source Accuracy on the frozen original inputs inside the v2 bank. Respan Span-02 respan/span-02-20261015; Hosted native; returned identities {"respan/span-02-20261015": 1934}; collected 2026-10-18 to 2026-10-31 America/Toronto; one primary response per input, no scored retries. Exact aggregate from the release statistics. Development benchmark, not an independently sampled test: 1,934 questions in 41 categories, the frozen 1,342-question bank plus 192 logical and 400 practical additions. The bank was built with model-guided development, and Jev plus retained Microsoft and Drex feedback are development exposure. Related variants and a single recorded primary response per input limit generalization. The scoring key stays private, so public checks cover coverage, timing and aggregate consistency but cannot rescore answers. Strict accuracy counts a wrong usable answer and an unsuccessful response in the same fixed denominator. Timing pools primary client times with the network included; providers, precision, concurrency, routes and dates differ, retained and new timings can span sessions, and quick failures can reduce all-primary latency. The source published a separate cost ledger on 1 November 2026, stored as decision-reference-v2-cost-per-1000, which supersedes its earlier “cost not consolidated” note. Older 1,342-only scores are not converted into this bank. Not ranked here. Checked / reported: 2026-11-02 · Samples: 1342 Snapshot: 2026-11-01 · Revision: f5fa9cfafccd97fcfbdb0336e14284faaa8d2d35 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
System 1 Decision Benchmark v2 · Median client time	228.4 ms	Setup & source Pooled primary complete-response client time, network included. Respan Span-02 respan/span-02-20261015; Hosted native; returned identities {"respan/span-02-20261015": 1934}; collected 2026-10-18 to 2026-10-31 America/Toronto; one primary response per input, no scored retries. Exact aggregate from the release statistics. Development benchmark, not an independently sampled test: 1,934 questions in 41 categories, the frozen 1,342-question bank plus 192 logical and 400 practical additions. The bank was built with model-guided development, and Jev plus retained Microsoft and Drex feedback are development exposure. Related variants and a single recorded primary response per input limit generalization. The scoring key stays private, so public checks cover coverage, timing and aggregate consistency but cannot rescore answers. Strict accuracy counts a wrong usable answer and an unsuccessful response in the same fixed denominator. Timing pools primary client times with the network included; providers, precision, concurrency, routes and dates differ, retained and new timings can span sessions, and quick failures can reduce all-primary latency. The source published a separate cost ledger on 1 November 2026, stored as decision-reference-v2-cost-per-1000, which supersedes its earlier “cost not consolidated” note. Older 1,342-only scores are not converted into this bank. Not ranked here. Checked / reported: 2026-11-02 · Samples: 1934 Snapshot: 2026-11-01 · Revision: f5fa9cfafccd97fcfbdb0336e14284faaa8d2d35 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
System 1 Decision Benchmark v2 · P95 client time	2068.9 ms	Setup & source Pooled primary complete-response client time, network included. Respan Span-02 respan/span-02-20261015; Hosted native; returned identities {"respan/span-02-20261015": 1934}; collected 2026-10-18 to 2026-10-31 America/Toronto; one primary response per input, no scored retries. Exact aggregate from the release statistics. Development benchmark, not an independently sampled test: 1,934 questions in 41 categories, the frozen 1,342-question bank plus 192 logical and 400 practical additions. The bank was built with model-guided development, and Jev plus retained Microsoft and Drex feedback are development exposure. Related variants and a single recorded primary response per input limit generalization. The scoring key stays private, so public checks cover coverage, timing and aggregate consistency but cannot rescore answers. Strict accuracy counts a wrong usable answer and an unsuccessful response in the same fixed denominator. Timing pools primary client times with the network included; providers, precision, concurrency, routes and dates differ, retained and new timings can span sessions, and quick failures can reduce all-primary latency. The source published a separate cost ledger on 1 November 2026, stored as decision-reference-v2-cost-per-1000, which supersedes its earlier “cost not consolidated” note. Older 1,342-only scores are not converted into this bank. Not ranked here. Checked / reported: 2026-11-02 · Samples: 1934 Snapshot: 2026-11-01 · Revision: f5fa9cfafccd97fcfbdb0336e14284faaa8d2d35 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
System 1 Decision Benchmark v2 · F01 · Lookup and filtering	90.698%	Setup & source Lookup and filtering: 43 questions. Respan Span-02 respan/span-02-20261015; Hosted native; returned identities {"respan/span-02-20261015": 1934}; collected 2026-10-18 to 2026-10-31 America/Toronto; one primary response per input, no scored retries. Exact aggregate from the release statistics. Development benchmark, not an independently sampled test: 1,934 questions in 41 categories, the frozen 1,342-question bank plus 192 logical and 400 practical additions. The bank was built with model-guided development, and Jev plus retained Microsoft and Drex feedback are development exposure. Related variants and a single recorded primary response per input limit generalization. The scoring key stays private, so public checks cover coverage, timing and aggregate consistency but cannot rescore answers. Strict accuracy counts a wrong usable answer and an unsuccessful response in the same fixed denominator. Timing pools primary client times with the network included; providers, precision, concurrency, routes and dates differ, retained and new timings can span sessions, and quick failures can reduce all-primary latency. The source published a separate cost ledger on 1 November 2026, stored as decision-reference-v2-cost-per-1000, which supersedes its earlier “cost not consolidated” note. Older 1,342-only scores are not converted into this bank. Not ranked here. Checked / reported: 2026-11-02 · Samples: 43 Snapshot: 2026-11-01 · Revision: f5fa9cfafccd97fcfbdb0336e14284faaa8d2d35 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
System 1 Decision Benchmark v2 · F02 · Forward conditionals	69.565%	Setup & source Forward conditionals: 46 questions. Respan Span-02 respan/span-02-20261015; Hosted native; returned identities {"respan/span-02-20261015": 1934}; collected 2026-10-18 to 2026-10-31 America/Toronto; one primary response per input, no scored retries. Exact aggregate from the release statistics. Development benchmark, not an independently sampled test: 1,934 questions in 41 categories, the frozen 1,342-question bank plus 192 logical and 400 practical additions. The bank was built with model-guided development, and Jev plus retained Microsoft and Drex feedback are development exposure. Related variants and a single recorded primary response per input limit generalization. The scoring key stays private, so public checks cover coverage, timing and aggregate consistency but cannot rescore answers. Strict accuracy counts a wrong usable answer and an unsuccessful response in the same fixed denominator. Timing pools primary client times with the network included; providers, precision, concurrency, routes and dates differ, retained and new timings can span sessions, and quick failures can reduce all-primary latency. The source published a separate cost ledger on 1 November 2026, stored as decision-reference-v2-cost-per-1000, which supersedes its earlier “cost not consolidated” note. Older 1,342-only scores are not converted into this bank. Not ranked here. Checked / reported: 2026-11-02 · Samples: 46 Snapshot: 2026-11-01 · Revision: f5fa9cfafccd97fcfbdb0336e14284faaa8d2d35 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
System 1 Decision Benchmark v2 · F03 · Necessary conditions	84.615%	Setup & source Necessary conditions: 52 questions. Respan Span-02 respan/span-02-20261015; Hosted native; returned identities {"respan/span-02-20261015": 1934}; collected 2026-10-18 to 2026-10-31 America/Toronto; one primary response per input, no scored retries. Exact aggregate from the release statistics. Development benchmark, not an independently sampled test: 1,934 questions in 41 categories, the frozen 1,342-question bank plus 192 logical and 400 practical additions. The bank was built with model-guided development, and Jev plus retained Microsoft and Drex feedback are development exposure. Related variants and a single recorded primary response per input limit generalization. The scoring key stays private, so public checks cover coverage, timing and aggregate consistency but cannot rescore answers. Strict accuracy counts a wrong usable answer and an unsuccessful response in the same fixed denominator. Timing pools primary client times with the network included; providers, precision, concurrency, routes and dates differ, retained and new timings can span sessions, and quick failures can reduce all-primary latency. The source published a separate cost ledger on 1 November 2026, stored as decision-reference-v2-cost-per-1000, which supersedes its earlier “cost not consolidated” note. Older 1,342-only scores are not converted into this bank. Not ranked here. Checked / reported: 2026-11-02 · Samples: 52 Snapshot: 2026-11-01 · Revision: f5fa9cfafccd97fcfbdb0336e14284faaa8d2d35 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
System 1 Decision Benchmark v2 · F04 · Boolean scope	92.593%	Setup & source Boolean scope: 54 questions. Respan Span-02 respan/span-02-20261015; Hosted native; returned identities {"respan/span-02-20261015": 1934}; collected 2026-10-18 to 2026-10-31 America/Toronto; one primary response per input, no scored retries. Exact aggregate from the release statistics. Development benchmark, not an independently sampled test: 1,934 questions in 41 categories, the frozen 1,342-question bank plus 192 logical and 400 practical additions. The bank was built with model-guided development, and Jev plus retained Microsoft and Drex feedback are development exposure. Related variants and a single recorded primary response per input limit generalization. The scoring key stays private, so public checks cover coverage, timing and aggregate consistency but cannot rescore answers. Strict accuracy counts a wrong usable answer and an unsuccessful response in the same fixed denominator. Timing pools primary client times with the network included; providers, precision, concurrency, routes and dates differ, retained and new timings can span sessions, and quick failures can reduce all-primary latency. The source published a separate cost ledger on 1 November 2026, stored as decision-reference-v2-cost-per-1000, which supersedes its earlier “cost not consolidated” note. Older 1,342-only scores are not converted into this bank. Not ranked here. Checked / reported: 2026-11-02 · Samples: 54 Snapshot: 2026-11-01 · Revision: f5fa9cfafccd97fcfbdb0336e14284faaa8d2d35 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
System 1 Decision Benchmark v2 · F05 · Quantifiers and sets	87.755%	Setup & source Quantifiers and sets: 49 questions. Respan Span-02 respan/span-02-20261015; Hosted native; returned identities {"respan/span-02-20261015": 1934}; collected 2026-10-18 to 2026-10-31 America/Toronto; one primary response per input, no scored retries. Exact aggregate from the release statistics. Development benchmark, not an independently sampled test: 1,934 questions in 41 categories, the frozen 1,342-question bank plus 192 logical and 400 practical additions. The bank was built with model-guided development, and Jev plus retained Microsoft and Drex feedback are development exposure. Related variants and a single recorded primary response per input limit generalization. The scoring key stays private, so public checks cover coverage, timing and aggregate consistency but cannot rescore answers. Strict accuracy counts a wrong usable answer and an unsuccessful response in the same fixed denominator. Timing pools primary client times with the network included; providers, precision, concurrency, routes and dates differ, retained and new timings can span sessions, and quick failures can reduce all-primary latency. The source published a separate cost ledger on 1 November 2026, stored as decision-reference-v2-cost-per-1000, which supersedes its earlier “cost not consolidated” note. Older 1,342-only scores are not converted into this bank. Not ranked here. Checked / reported: 2026-11-02 · Samples: 49 Snapshot: 2026-11-01 · Revision: f5fa9cfafccd97fcfbdb0336e14284faaa8d2d35 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
System 1 Decision Benchmark v2 · F06 · Comparisons	95.000%	Setup & source Comparisons: 40 questions. Respan Span-02 respan/span-02-20261015; Hosted native; returned identities {"respan/span-02-20261015": 1934}; collected 2026-10-18 to 2026-10-31 America/Toronto; one primary response per input, no scored retries. Exact aggregate from the release statistics. Development benchmark, not an independently sampled test: 1,934 questions in 41 categories, the frozen 1,342-question bank plus 192 logical and 400 practical additions. The bank was built with model-guided development, and Jev plus retained Microsoft and Drex feedback are development exposure. Related variants and a single recorded primary response per input limit generalization. The scoring key stays private, so public checks cover coverage, timing and aggregate consistency but cannot rescore answers. Strict accuracy counts a wrong usable answer and an unsuccessful response in the same fixed denominator. Timing pools primary client times with the network included; providers, precision, concurrency, routes and dates differ, retained and new timings can span sessions, and quick failures can reduce all-primary latency. The source published a separate cost ledger on 1 November 2026, stored as decision-reference-v2-cost-per-1000, which supersedes its earlier “cost not consolidated” note. Older 1,342-only scores are not converted into this bank. Not ranked here. Checked / reported: 2026-11-02 · Samples: 40 Snapshot: 2026-11-01 · Revision: f5fa9cfafccd97fcfbdb0336e14284faaa8d2d35 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
System 1 Decision Benchmark v2 · F07 · Scheduling	76.316%	Setup & source Scheduling: 76 questions. Respan Span-02 respan/span-02-20261015; Hosted native; returned identities {"respan/span-02-20261015": 1934}; collected 2026-10-18 to 2026-10-31 America/Toronto; one primary response per input, no scored retries. Exact aggregate from the release statistics. Development benchmark, not an independently sampled test: 1,934 questions in 41 categories, the frozen 1,342-question bank plus 192 logical and 400 practical additions. The bank was built with model-guided development, and Jev plus retained Microsoft and Drex feedback are development exposure. Related variants and a single recorded primary response per input limit generalization. The scoring key stays private, so public checks cover coverage, timing and aggregate consistency but cannot rescore answers. Strict accuracy counts a wrong usable answer and an unsuccessful response in the same fixed denominator. Timing pools primary client times with the network included; providers, precision, concurrency, routes and dates differ, retained and new timings can span sessions, and quick failures can reduce all-primary latency. The source published a separate cost ledger on 1 November 2026, stored as decision-reference-v2-cost-per-1000, which supersedes its earlier “cost not consolidated” note. Older 1,342-only scores are not converted into this bank. Not ranked here. Checked / reported: 2026-11-02 · Samples: 76 Snapshot: 2026-11-01 · Revision: f5fa9cfafccd97fcfbdb0336e14284faaa8d2d35 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
System 1 Decision Benchmark v2 · F08 · One-to-one assignments	94.030%	Setup & source One-to-one assignments: 67 questions. Respan Span-02 respan/span-02-20261015; Hosted native; returned identities {"respan/span-02-20261015": 1934}; collected 2026-10-18 to 2026-10-31 America/Toronto; one primary response per input, no scored retries. Exact aggregate from the release statistics. Development benchmark, not an independently sampled test: 1,934 questions in 41 categories, the frozen 1,342-question bank plus 192 logical and 400 practical additions. The bank was built with model-guided development, and Jev plus retained Microsoft and Drex feedback are development exposure. Related variants and a single recorded primary response per input limit generalization. The scoring key stays private, so public checks cover coverage, timing and aggregate consistency but cannot rescore answers. Strict accuracy counts a wrong usable answer and an unsuccessful response in the same fixed denominator. Timing pools primary client times with the network included; providers, precision, concurrency, routes and dates differ, retained and new timings can span sessions, and quick failures can reduce all-primary latency. The source published a separate cost ledger on 1 November 2026, stored as decision-reference-v2-cost-per-1000, which supersedes its earlier “cost not consolidated” note. Older 1,342-only scores are not converted into this bank. Not ranked here. Checked / reported: 2026-11-02 · Samples: 67 Snapshot: 2026-11-01 · Revision: f5fa9cfafccd97fcfbdb0336e14284faaa8d2d35 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
System 1 Decision Benchmark v2 · F09 · Exact truth counting	64.789%	Setup & source Exact truth counting: 71 questions. Respan Span-02 respan/span-02-20261015; Hosted native; returned identities {"respan/span-02-20261015": 1934}; collected 2026-10-18 to 2026-10-31 America/Toronto; one primary response per input, no scored retries. Exact aggregate from the release statistics. Development benchmark, not an independently sampled test: 1,934 questions in 41 categories, the frozen 1,342-question bank plus 192 logical and 400 practical additions. The bank was built with model-guided development, and Jev plus retained Microsoft and Drex feedback are development exposure. Related variants and a single recorded primary response per input limit generalization. The scoring key stays private, so public checks cover coverage, timing and aggregate consistency but cannot rescore answers. Strict accuracy counts a wrong usable answer and an unsuccessful response in the same fixed denominator. Timing pools primary client times with the network included; providers, precision, concurrency, routes and dates differ, retained and new timings can span sessions, and quick failures can reduce all-primary latency. The source published a separate cost ledger on 1 November 2026, stored as decision-reference-v2-cost-per-1000, which supersedes its earlier “cost not consolidated” note. Older 1,342-only scores are not converted into this bank. Not ranked here. Checked / reported: 2026-11-02 · Samples: 71 Snapshot: 2026-11-01 · Revision: f5fa9cfafccd97fcfbdb0336e14284faaa8d2d35 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
System 1 Decision Benchmark v2 · F10 · Truth-teller consistency	74.324%	Setup & source Truth-teller consistency: 74 questions. Respan Span-02 respan/span-02-20261015; Hosted native; returned identities {"respan/span-02-20261015": 1934}; collected 2026-10-18 to 2026-10-31 America/Toronto; one primary response per input, no scored retries. Exact aggregate from the release statistics. Development benchmark, not an independently sampled test: 1,934 questions in 41 categories, the frozen 1,342-question bank plus 192 logical and 400 practical additions. The bank was built with model-guided development, and Jev plus retained Microsoft and Drex feedback are development exposure. Related variants and a single recorded primary response per input limit generalization. The scoring key stays private, so public checks cover coverage, timing and aggregate consistency but cannot rescore answers. Strict accuracy counts a wrong usable answer and an unsuccessful response in the same fixed denominator. Timing pools primary client times with the network included; providers, precision, concurrency, routes and dates differ, retained and new timings can span sessions, and quick failures can reduce all-primary latency. The source published a separate cost ledger on 1 November 2026, stored as decision-reference-v2-cost-per-1000, which supersedes its earlier “cost not consolidated” note. Older 1,342-only scores are not converted into this bank. Not ranked here. Checked / reported: 2026-11-02 · Samples: 74 Snapshot: 2026-11-01 · Revision: f5fa9cfafccd97fcfbdb0336e14284faaa8d2d35 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
System 1 Decision Benchmark v2 · F11 · Sufficiency and inconsistency	87.805%	Setup & source Sufficiency and inconsistency: 41 questions. Respan Span-02 respan/span-02-20261015; Hosted native; returned identities {"respan/span-02-20261015": 1934}; collected 2026-10-18 to 2026-10-31 America/Toronto; one primary response per input, no scored retries. Exact aggregate from the release statistics. Development benchmark, not an independently sampled test: 1,934 questions in 41 categories, the frozen 1,342-question bank plus 192 logical and 400 practical additions. The bank was built with model-guided development, and Jev plus retained Microsoft and Drex feedback are development exposure. Related variants and a single recorded primary response per input limit generalization. The scoring key stays private, so public checks cover coverage, timing and aggregate consistency but cannot rescore answers. Strict accuracy counts a wrong usable answer and an unsuccessful response in the same fixed denominator. Timing pools primary client times with the network included; providers, precision, concurrency, routes and dates differ, retained and new timings can span sessions, and quick failures can reduce all-primary latency. The source published a separate cost ledger on 1 November 2026, stored as decision-reference-v2-cost-per-1000, which supersedes its earlier “cost not consolidated” note. Older 1,342-only scores are not converted into this bank. Not ranked here. Checked / reported: 2026-11-02 · Samples: 41 Snapshot: 2026-11-01 · Revision: f5fa9cfafccd97fcfbdb0336e14284faaa8d2d35 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
System 1 Decision Benchmark v2 · F12 · Object swaps	53.846%	Setup & source Object swaps: 52 questions. Respan Span-02 respan/span-02-20261015; Hosted native; returned identities {"respan/span-02-20261015": 1934}; collected 2026-10-18 to 2026-10-31 America/Toronto; one primary response per input, no scored retries. Exact aggregate from the release statistics. Development benchmark, not an independently sampled test: 1,934 questions in 41 categories, the frozen 1,342-question bank plus 192 logical and 400 practical additions. The bank was built with model-guided development, and Jev plus retained Microsoft and Drex feedback are development exposure. Related variants and a single recorded primary response per input limit generalization. The scoring key stays private, so public checks cover coverage, timing and aggregate consistency but cannot rescore answers. Strict accuracy counts a wrong usable answer and an unsuccessful response in the same fixed denominator. Timing pools primary client times with the network included; providers, precision, concurrency, routes and dates differ, retained and new timings can span sessions, and quick failures can reduce all-primary latency. The source published a separate cost ledger on 1 November 2026, stored as decision-reference-v2-cost-per-1000, which supersedes its earlier “cost not consolidated” note. Older 1,342-only scores are not converted into this bank. Not ranked here. Checked / reported: 2026-11-02 · Samples: 52 Snapshot: 2026-11-01 · Revision: f5fa9cfafccd97fcfbdb0336e14284faaa8d2d35 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
System 1 Decision Benchmark v2 · F13 · Sequential procedures	68.750%	Setup & source Sequential procedures: 48 questions. Respan Span-02 respan/span-02-20261015; Hosted native; returned identities {"respan/span-02-20261015": 1934}; collected 2026-10-18 to 2026-10-31 America/Toronto; one primary response per input, no scored retries. Exact aggregate from the release statistics. Development benchmark, not an independently sampled test: 1,934 questions in 41 categories, the frozen 1,342-question bank plus 192 logical and 400 practical additions. The bank was built with model-guided development, and Jev plus retained Microsoft and Drex feedback are development exposure. Related variants and a single recorded primary response per input limit generalization. The scoring key stays private, so public checks cover coverage, timing and aggregate consistency but cannot rescore answers. Strict accuracy counts a wrong usable answer and an unsuccessful response in the same fixed denominator. Timing pools primary client times with the network included; providers, precision, concurrency, routes and dates differ, retained and new timings can span sessions, and quick failures can reduce all-primary latency. The source published a separate cost ledger on 1 November 2026, stored as decision-reference-v2-cost-per-1000, which supersedes its earlier “cost not consolidated” note. Older 1,342-only scores are not converted into this bank. Not ranked here. Checked / reported: 2026-11-02 · Samples: 48 Snapshot: 2026-11-01 · Revision: f5fa9cfafccd97fcfbdb0336e14284faaa8d2d35 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
System 1 Decision Benchmark v2 · F14 · Temporal intervals	95.238%	Setup & source Temporal intervals: 42 questions. Respan Span-02 respan/span-02-20261015; Hosted native; returned identities {"respan/span-02-20261015": 1934}; collected 2026-10-18 to 2026-10-31 America/Toronto; one primary response per input, no scored retries. Exact aggregate from the release statistics. Development benchmark, not an independently sampled test: 1,934 questions in 41 categories, the frozen 1,342-question bank plus 192 logical and 400 practical additions. The bank was built with model-guided development, and Jev plus retained Microsoft and Drex feedback are development exposure. Related variants and a single recorded primary response per input limit generalization. The scoring key stays private, so public checks cover coverage, timing and aggregate consistency but cannot rescore answers. Strict accuracy counts a wrong usable answer and an unsuccessful response in the same fixed denominator. Timing pools primary client times with the network included; providers, precision, concurrency, routes and dates differ, retained and new timings can span sessions, and quick failures can reduce all-primary latency. The source published a separate cost ledger on 1 November 2026, stored as decision-reference-v2-cost-per-1000, which supersedes its earlier “cost not consolidated” note. Older 1,342-only scores are not converted into this bank. Not ranked here. Checked / reported: 2026-11-02 · Samples: 42 Snapshot: 2026-11-01 · Revision: f5fa9cfafccd97fcfbdb0336e14284faaa8d2d35 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
System 1 Decision Benchmark v2 · F15 · Spatial tracking	85.714%	Setup & source Spatial tracking: 49 questions. Respan Span-02 respan/span-02-20261015; Hosted native; returned identities {"respan/span-02-20261015": 1934}; collected 2026-10-18 to 2026-10-31 America/Toronto; one primary response per input, no scored retries. Exact aggregate from the release statistics. Development benchmark, not an independently sampled test: 1,934 questions in 41 categories, the frozen 1,342-question bank plus 192 logical and 400 practical additions. The bank was built with model-guided development, and Jev plus retained Microsoft and Drex feedback are development exposure. Related variants and a single recorded primary response per input limit generalization. The scoring key stays private, so public checks cover coverage, timing and aggregate consistency but cannot rescore answers. Strict accuracy counts a wrong usable answer and an unsuccessful response in the same fixed denominator. Timing pools primary client times with the network included; providers, precision, concurrency, routes and dates differ, retained and new timings can span sessions, and quick failures can reduce all-primary latency. The source published a separate cost ledger on 1 November 2026, stored as decision-reference-v2-cost-per-1000, which supersedes its earlier “cost not consolidated” note. Older 1,342-only scores are not converted into this bank. Not ranked here. Checked / reported: 2026-11-02 · Samples: 49 Snapshot: 2026-11-01 · Revision: f5fa9cfafccd97fcfbdb0336e14284faaa8d2d35 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
System 1 Decision Benchmark v2 · F16 · Reference binding	84.000%	Setup & source Reference binding: 50 questions. Respan Span-02 respan/span-02-20261015; Hosted native; returned identities {"respan/span-02-20261015": 1934}; collected 2026-10-18 to 2026-10-31 America/Toronto; one primary response per input, no scored retries. Exact aggregate from the release statistics. Development benchmark, not an independently sampled test: 1,934 questions in 41 categories, the frozen 1,342-question bank plus 192 logical and 400 practical additions. The bank was built with model-guided development, and Jev plus retained Microsoft and Drex feedback are development exposure. Related variants and a single recorded primary response per input limit generalization. The scoring key stays private, so public checks cover coverage, timing and aggregate consistency but cannot rescore answers. Strict accuracy counts a wrong usable answer and an unsuccessful response in the same fixed denominator. Timing pools primary client times with the network included; providers, precision, concurrency, routes and dates differ, retained and new timings can span sessions, and quick failures can reduce all-primary latency. The source published a separate cost ledger on 1 November 2026, stored as decision-reference-v2-cost-per-1000, which supersedes its earlier “cost not consolidated” note. Older 1,342-only scores are not converted into this bank. Not ranked here. Checked / reported: 2026-11-02 · Samples: 50 Snapshot: 2026-11-01 · Revision: f5fa9cfafccd97fcfbdb0336e14284faaa8d2d35 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
System 1 Decision Benchmark v2 · F17 · Beliefs and perspectives	82.000%	Setup & source Beliefs and perspectives: 50 questions. Respan Span-02 respan/span-02-20261015; Hosted native; returned identities {"respan/span-02-20261015": 1934}; collected 2026-10-18 to 2026-10-31 America/Toronto; one primary response per input, no scored retries. Exact aggregate from the release statistics. Development benchmark, not an independently sampled test: 1,934 questions in 41 categories, the frozen 1,342-question bank plus 192 logical and 400 practical additions. The bank was built with model-guided development, and Jev plus retained Microsoft and Drex feedback are development exposure. Related variants and a single recorded primary response per input limit generalization. The scoring key stays private, so public checks cover coverage, timing and aggregate consistency but cannot rescore answers. Strict accuracy counts a wrong usable answer and an unsuccessful response in the same fixed denominator. Timing pools primary client times with the network included; providers, precision, concurrency, routes and dates differ, retained and new timings can span sessions, and quick failures can reduce all-primary latency. The source published a separate cost ledger on 1 November 2026, stored as decision-reference-v2-cost-per-1000, which supersedes its earlier “cost not consolidated” note. Older 1,342-only scores are not converted into this bank. Not ranked here. Checked / reported: 2026-11-02 · Samples: 50 Snapshot: 2026-11-01 · Revision: f5fa9cfafccd97fcfbdb0336e14284faaa8d2d35 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
System 1 Decision Benchmark v2 · F18 · Causal intervention	94.000%	Setup & source Causal intervention: 50 questions. Respan Span-02 respan/span-02-20261015; Hosted native; returned identities {"respan/span-02-20261015": 1934}; collected 2026-10-18 to 2026-10-31 America/Toronto; one primary response per input, no scored retries. Exact aggregate from the release statistics. Development benchmark, not an independently sampled test: 1,934 questions in 41 categories, the frozen 1,342-question bank plus 192 logical and 400 practical additions. The bank was built with model-guided development, and Jev plus retained Microsoft and Drex feedback are development exposure. Related variants and a single recorded primary response per input limit generalization. The scoring key stays private, so public checks cover coverage, timing and aggregate consistency but cannot rescore answers. Strict accuracy counts a wrong usable answer and an unsuccessful response in the same fixed denominator. Timing pools primary client times with the network included; providers, precision, concurrency, routes and dates differ, retained and new timings can span sessions, and quick failures can reduce all-primary latency. The source published a separate cost ledger on 1 November 2026, stored as decision-reference-v2-cost-per-1000, which supersedes its earlier “cost not consolidated” note. Older 1,342-only scores are not converted into this bank. Not ranked here. Checked / reported: 2026-11-02 · Samples: 50 Snapshot: 2026-11-01 · Revision: f5fa9cfafccd97fcfbdb0336e14284faaa8d2d35 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
System 1 Decision Benchmark v2 · F19 · Exceptions and priorities	91.379%	Setup & source Exceptions and priorities: 58 questions. Respan Span-02 respan/span-02-20261015; Hosted native; returned identities {"respan/span-02-20261015": 1934}; collected 2026-10-18 to 2026-10-31 America/Toronto; one primary response per input, no scored retries. Exact aggregate from the release statistics. Development benchmark, not an independently sampled test: 1,934 questions in 41 categories, the frozen 1,342-question bank plus 192 logical and 400 practical additions. The bank was built with model-guided development, and Jev plus retained Microsoft and Drex feedback are development exposure. Related variants and a single recorded primary response per input limit generalization. The scoring key stays private, so public checks cover coverage, timing and aggregate consistency but cannot rescore answers. Strict accuracy counts a wrong usable answer and an unsuccessful response in the same fixed denominator. Timing pools primary client times with the network included; providers, precision, concurrency, routes and dates differ, retained and new timings can span sessions, and quick failures can reduce all-primary latency. The source published a separate cost ledger on 1 November 2026, stored as decision-reference-v2-cost-per-1000, which supersedes its earlier “cost not consolidated” note. Older 1,342-only scores are not converted into this bank. Not ranked here. Checked / reported: 2026-11-02 · Samples: 58 Snapshot: 2026-11-01 · Revision: f5fa9cfafccd97fcfbdb0336e14284faaa8d2d35 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
System 1 Decision Benchmark v2 · F20 · Planning	90.698%	Setup & source Planning: 43 questions. Respan Span-02 respan/span-02-20261015; Hosted native; returned identities {"respan/span-02-20261015": 1934}; collected 2026-10-18 to 2026-10-31 America/Toronto; one primary response per input, no scored retries. Exact aggregate from the release statistics. Development benchmark, not an independently sampled test: 1,934 questions in 41 categories, the frozen 1,342-question bank plus 192 logical and 400 practical additions. The bank was built with model-guided development, and Jev plus retained Microsoft and Drex feedback are development exposure. Related variants and a single recorded primary response per input limit generalization. The scoring key stays private, so public checks cover coverage, timing and aggregate consistency but cannot rescore answers. Strict accuracy counts a wrong usable answer and an unsuccessful response in the same fixed denominator. Timing pools primary client times with the network included; providers, precision, concurrency, routes and dates differ, retained and new timings can span sessions, and quick failures can reduce all-primary latency. The source published a separate cost ledger on 1 November 2026, stored as decision-reference-v2-cost-per-1000, which supersedes its earlier “cost not consolidated” note. Older 1,342-only scores are not converted into this bank. Not ranked here. Checked / reported: 2026-11-02 · Samples: 43 Snapshot: 2026-11-01 · Revision: f5fa9cfafccd97fcfbdb0336e14284faaa8d2d35 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
System 1 Decision Benchmark v2 · F21 · Directed reachability	93.023%	Setup & source Directed reachability: 43 questions. Respan Span-02 respan/span-02-20261015; Hosted native; returned identities {"respan/span-02-20261015": 1934}; collected 2026-10-18 to 2026-10-31 America/Toronto; one primary response per input, no scored retries. Exact aggregate from the release statistics. Development benchmark, not an independently sampled test: 1,934 questions in 41 categories, the frozen 1,342-question bank plus 192 logical and 400 practical additions. The bank was built with model-guided development, and Jev plus retained Microsoft and Drex feedback are development exposure. Related variants and a single recorded primary response per input limit generalization. The scoring key stays private, so public checks cover coverage, timing and aggregate consistency but cannot rescore answers. Strict accuracy counts a wrong usable answer and an unsuccessful response in the same fixed denominator. Timing pools primary client times with the network included; providers, precision, concurrency, routes and dates differ, retained and new timings can span sessions, and quick failures can reduce all-primary latency. The source published a separate cost ledger on 1 November 2026, stored as decision-reference-v2-cost-per-1000, which supersedes its earlier “cost not consolidated” note. Older 1,342-only scores are not converted into this bank. Not ranked here. Checked / reported: 2026-11-02 · Samples: 43 Snapshot: 2026-11-01 · Revision: f5fa9cfafccd97fcfbdb0336e14284faaa8d2d35 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
System 1 Decision Benchmark v2 · F22 · Preference ordering	96.078%	Setup & source Preference ordering: 51 questions. Respan Span-02 respan/span-02-20261015; Hosted native; returned identities {"respan/span-02-20261015": 1934}; collected 2026-10-18 to 2026-10-31 America/Toronto; one primary response per input, no scored retries. Exact aggregate from the release statistics. Development benchmark, not an independently sampled test: 1,934 questions in 41 categories, the frozen 1,342-question bank plus 192 logical and 400 practical additions. The bank was built with model-guided development, and Jev plus retained Microsoft and Drex feedback are development exposure. Related variants and a single recorded primary response per input limit generalization. The scoring key stays private, so public checks cover coverage, timing and aggregate consistency but cannot rescore answers. Strict accuracy counts a wrong usable answer and an unsuccessful response in the same fixed denominator. Timing pools primary client times with the network included; providers, precision, concurrency, routes and dates differ, retained and new timings can span sessions, and quick failures can reduce all-primary latency. The source published a separate cost ledger on 1 November 2026, stored as decision-reference-v2-cost-per-1000, which supersedes its earlier “cost not consolidated” note. Older 1,342-only scores are not converted into this bank. Not ranked here. Checked / reported: 2026-11-02 · Samples: 51 Snapshot: 2026-11-01 · Revision: f5fa9cfafccd97fcfbdb0336e14284faaa8d2d35 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
System 1 Decision Benchmark v2 · F23 · Rule counterexamples	98.077%	Setup & source Rule counterexamples: 52 questions. Respan Span-02 respan/span-02-20261015; Hosted native; returned identities {"respan/span-02-20261015": 1934}; collected 2026-10-18 to 2026-10-31 America/Toronto; one primary response per input, no scored retries. Exact aggregate from the release statistics. Development benchmark, not an independently sampled test: 1,934 questions in 41 categories, the frozen 1,342-question bank plus 192 logical and 400 practical additions. The bank was built with model-guided development, and Jev plus retained Microsoft and Drex feedback are development exposure. Related variants and a single recorded primary response per input limit generalization. The scoring key stays private, so public checks cover coverage, timing and aggregate consistency but cannot rescore answers. Strict accuracy counts a wrong usable answer and an unsuccessful response in the same fixed denominator. Timing pools primary client times with the network included; providers, precision, concurrency, routes and dates differ, retained and new timings can span sessions, and quick failures can reduce all-primary latency. The source published a separate cost ledger on 1 November 2026, stored as decision-reference-v2-cost-per-1000, which supersedes its earlier “cost not consolidated” note. Older 1,342-only scores are not converted into this bank. Not ranked here. Checked / reported: 2026-11-02 · Samples: 52 Snapshot: 2026-11-01 · Revision: f5fa9cfafccd97fcfbdb0336e14284faaa8d2d35 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
System 1 Decision Benchmark v2 · F24 · Authority and stale records	88.372%	Setup & source Authority and stale records: 43 questions. Respan Span-02 respan/span-02-20261015; Hosted native; returned identities {"respan/span-02-20261015": 1934}; collected 2026-10-18 to 2026-10-31 America/Toronto; one primary response per input, no scored retries. Exact aggregate from the release statistics. Development benchmark, not an independently sampled test: 1,934 questions in 41 categories, the frozen 1,342-question bank plus 192 logical and 400 practical additions. The bank was built with model-guided development, and Jev plus retained Microsoft and Drex feedback are development exposure. Related variants and a single recorded primary response per input limit generalization. The scoring key stays private, so public checks cover coverage, timing and aggregate consistency but cannot rescore answers. Strict accuracy counts a wrong usable answer and an unsuccessful response in the same fixed denominator. Timing pools primary client times with the network included; providers, precision, concurrency, routes and dates differ, retained and new timings can span sessions, and quick failures can reduce all-primary latency. The source published a separate cost ledger on 1 November 2026, stored as decision-reference-v2-cost-per-1000, which supersedes its earlier “cost not consolidated” note. Older 1,342-only scores are not converted into this bank. Not ranked here. Checked / reported: 2026-11-02 · Samples: 43 Snapshot: 2026-11-01 · Revision: f5fa9cfafccd97fcfbdb0336e14284faaa8d2d35 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
System 1 Decision Benchmark v2 · F25 · Fault diagnosis	92.000%	Setup & source Fault diagnosis: 50 questions. Respan Span-02 respan/span-02-20261015; Hosted native; returned identities {"respan/span-02-20261015": 1934}; collected 2026-10-18 to 2026-10-31 America/Toronto; one primary response per input, no scored retries. Exact aggregate from the release statistics. Development benchmark, not an independently sampled test: 1,934 questions in 41 categories, the frozen 1,342-question bank plus 192 logical and 400 practical additions. The bank was built with model-guided development, and Jev plus retained Microsoft and Drex feedback are development exposure. Related variants and a single recorded primary response per input limit generalization. The scoring key stays private, so public checks cover coverage, timing and aggregate consistency but cannot rescore answers. Strict accuracy counts a wrong usable answer and an unsuccessful response in the same fixed denominator. Timing pools primary client times with the network included; providers, precision, concurrency, routes and dates differ, retained and new timings can span sessions, and quick failures can reduce all-primary latency. The source published a separate cost ledger on 1 November 2026, stored as decision-reference-v2-cost-per-1000, which supersedes its earlier “cost not consolidated” note. Older 1,342-only scores are not converted into this bank. Not ranked here. Checked / reported: 2026-11-02 · Samples: 50 Snapshot: 2026-11-01 · Revision: f5fa9cfafccd97fcfbdb0336e14284faaa8d2d35 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
System 1 Decision Benchmark v2 · F26 · Probabilistic risk	65.000%	Setup & source Probabilistic risk: 40 questions. Respan Span-02 respan/span-02-20261015; Hosted native; returned identities {"respan/span-02-20261015": 1934}; collected 2026-10-18 to 2026-10-31 America/Toronto; one primary response per input, no scored retries. Exact aggregate from the release statistics. Development benchmark, not an independently sampled test: 1,934 questions in 41 categories, the frozen 1,342-question bank plus 192 logical and 400 practical additions. The bank was built with model-guided development, and Jev plus retained Microsoft and Drex feedback are development exposure. Related variants and a single recorded primary response per input limit generalization. The scoring key stays private, so public checks cover coverage, timing and aggregate consistency but cannot rescore answers. Strict accuracy counts a wrong usable answer and an unsuccessful response in the same fixed denominator. Timing pools primary client times with the network included; providers, precision, concurrency, routes and dates differ, retained and new timings can span sessions, and quick failures can reduce all-primary latency. The source published a separate cost ledger on 1 November 2026, stored as decision-reference-v2-cost-per-1000, which supersedes its earlier “cost not consolidated” note. Older 1,342-only scores are not converted into this bank. Not ranked here. Checked / reported: 2026-11-02 · Samples: 40 Snapshot: 2026-11-01 · Revision: f5fa9cfafccd97fcfbdb0336e14284faaa8d2d35 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
System 1 Decision Benchmark v2 · F27 · Value of information	52.500%	Setup & source Value of information: 40 questions. Respan Span-02 respan/span-02-20261015; Hosted native; returned identities {"respan/span-02-20261015": 1934}; collected 2026-10-18 to 2026-10-31 America/Toronto; one primary response per input, no scored retries. Exact aggregate from the release statistics. Development benchmark, not an independently sampled test: 1,934 questions in 41 categories, the frozen 1,342-question bank plus 192 logical and 400 practical additions. The bank was built with model-guided development, and Jev plus retained Microsoft and Drex feedback are development exposure. Related variants and a single recorded primary response per input limit generalization. The scoring key stays private, so public checks cover coverage, timing and aggregate consistency but cannot rescore answers. Strict accuracy counts a wrong usable answer and an unsuccessful response in the same fixed denominator. Timing pools primary client times with the network included; providers, precision, concurrency, routes and dates differ, retained and new timings can span sessions, and quick failures can reduce all-primary latency. The source published a separate cost ledger on 1 November 2026, stored as decision-reference-v2-cost-per-1000, which supersedes its earlier “cost not consolidated” note. Older 1,342-only scores are not converted into this bank. Not ranked here. Checked / reported: 2026-11-02 · Samples: 40 Snapshot: 2026-11-01 · Revision: f5fa9cfafccd97fcfbdb0336e14284faaa8d2d35 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
System 1 Decision Benchmark v2 · F28 · Robust decisions without a prior	40.000%	Setup & source Robust decisions without a prior: 40 questions. Respan Span-02 respan/span-02-20261015; Hosted native; returned identities {"respan/span-02-20261015": 1934}; collected 2026-10-18 to 2026-10-31 America/Toronto; one primary response per input, no scored retries. Exact aggregate from the release statistics. Development benchmark, not an independently sampled test: 1,934 questions in 41 categories, the frozen 1,342-question bank plus 192 logical and 400 practical additions. The bank was built with model-guided development, and Jev plus retained Microsoft and Drex feedback are development exposure. Related variants and a single recorded primary response per input limit generalization. The scoring key stays private, so public checks cover coverage, timing and aggregate consistency but cannot rescore answers. Strict accuracy counts a wrong usable answer and an unsuccessful response in the same fixed denominator. Timing pools primary client times with the network included; providers, precision, concurrency, routes and dates differ, retained and new timings can span sessions, and quick failures can reduce all-primary latency. The source published a separate cost ledger on 1 November 2026, stored as decision-reference-v2-cost-per-1000, which supersedes its earlier “cost not consolidated” note. Older 1,342-only scores are not converted into this bank. Not ranked here. Checked / reported: 2026-11-02 · Samples: 40 Snapshot: 2026-11-01 · Revision: f5fa9cfafccd97fcfbdb0336e14284faaa8d2d35 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
System 1 Decision Benchmark v2 · F29 · Strategic decisions	52.500%	Setup & source Strategic decisions: 40 questions. Respan Span-02 respan/span-02-20261015; Hosted native; returned identities {"respan/span-02-20261015": 1934}; collected 2026-10-18 to 2026-10-31 America/Toronto; one primary response per input, no scored retries. Exact aggregate from the release statistics. Development benchmark, not an independently sampled test: 1,934 questions in 41 categories, the frozen 1,342-question bank plus 192 logical and 400 practical additions. The bank was built with model-guided development, and Jev plus retained Microsoft and Drex feedback are development exposure. Related variants and a single recorded primary response per input limit generalization. The scoring key stays private, so public checks cover coverage, timing and aggregate consistency but cannot rescore answers. Strict accuracy counts a wrong usable answer and an unsuccessful response in the same fixed denominator. Timing pools primary client times with the network included; providers, precision, concurrency, routes and dates differ, retained and new timings can span sessions, and quick failures can reduce all-primary latency. The source published a separate cost ledger on 1 November 2026, stored as decision-reference-v2-cost-per-1000, which supersedes its earlier “cost not consolidated” note. Older 1,342-only scores are not converted into this bank. Not ranked here. Checked / reported: 2026-11-02 · Samples: 40 Snapshot: 2026-11-01 · Revision: f5fa9cfafccd97fcfbdb0336e14284faaa8d2d35 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
System 1 Decision Benchmark v2 · F30 · Concurrent-history consistency	60.000%	Setup & source Concurrent-history consistency: 40 questions. Respan Span-02 respan/span-02-20261015; Hosted native; returned identities {"respan/span-02-20261015": 1934}; collected 2026-10-18 to 2026-10-31 America/Toronto; one primary response per input, no scored retries. Exact aggregate from the release statistics. Development benchmark, not an independently sampled test: 1,934 questions in 41 categories, the frozen 1,342-question bank plus 192 logical and 400 practical additions. The bank was built with model-guided development, and Jev plus retained Microsoft and Drex feedback are development exposure. Related variants and a single recorded primary response per input limit generalization. The scoring key stays private, so public checks cover coverage, timing and aggregate consistency but cannot rescore answers. Strict accuracy counts a wrong usable answer and an unsuccessful response in the same fixed denominator. Timing pools primary client times with the network included; providers, precision, concurrency, routes and dates differ, retained and new timings can span sessions, and quick failures can reduce all-primary latency. The source published a separate cost ledger on 1 November 2026, stored as decision-reference-v2-cost-per-1000, which supersedes its earlier “cost not consolidated” note. Older 1,342-only scores are not converted into this bank. Not ranked here. Checked / reported: 2026-11-02 · Samples: 40 Snapshot: 2026-11-01 · Revision: f5fa9cfafccd97fcfbdb0336e14284faaa8d2d35 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
System 1 Decision Benchmark v2 · F31 · Information-flow security	52.500%	Setup & source Information-flow security: 40 questions. Respan Span-02 respan/span-02-20261015; Hosted native; returned identities {"respan/span-02-20261015": 1934}; collected 2026-10-18 to 2026-10-31 America/Toronto; one primary response per input, no scored retries. Exact aggregate from the release statistics. Development benchmark, not an independently sampled test: 1,934 questions in 41 categories, the frozen 1,342-question bank plus 192 logical and 400 practical additions. The bank was built with model-guided development, and Jev plus retained Microsoft and Drex feedback are development exposure. Related variants and a single recorded primary response per input limit generalization. The scoring key stays private, so public checks cover coverage, timing and aggregate consistency but cannot rescore answers. Strict accuracy counts a wrong usable answer and an unsuccessful response in the same fixed denominator. Timing pools primary client times with the network included; providers, precision, concurrency, routes and dates differ, retained and new timings can span sessions, and quick failures can reduce all-primary latency. The source published a separate cost ledger on 1 November 2026, stored as decision-reference-v2-cost-per-1000, which supersedes its earlier “cost not consolidated” note. Older 1,342-only scores are not converted into this bank. Not ranked here. Checked / reported: 2026-11-02 · Samples: 40 Snapshot: 2026-11-01 · Revision: f5fa9cfafccd97fcfbdb0336e14284faaa8d2d35 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
System 1 Decision Benchmark v2 · P01 · Understanding requests and routing	62.500%	Setup & source Understanding requests and routing: 40 questions. Respan Span-02 respan/span-02-20261015; Hosted native; returned identities {"respan/span-02-20261015": 1934}; collected 2026-10-18 to 2026-10-31 America/Toronto; one primary response per input, no scored retries. Exact aggregate from the release statistics. Development benchmark, not an independently sampled test: 1,934 questions in 41 categories, the frozen 1,342-question bank plus 192 logical and 400 practical additions. The bank was built with model-guided development, and Jev plus retained Microsoft and Drex feedback are development exposure. Related variants and a single recorded primary response per input limit generalization. The scoring key stays private, so public checks cover coverage, timing and aggregate consistency but cannot rescore answers. Strict accuracy counts a wrong usable answer and an unsuccessful response in the same fixed denominator. Timing pools primary client times with the network included; providers, precision, concurrency, routes and dates differ, retained and new timings can span sessions, and quick failures can reduce all-primary latency. The source published a separate cost ledger on 1 November 2026, stored as decision-reference-v2-cost-per-1000, which supersedes its earlier “cost not consolidated” note. Older 1,342-only scores are not converted into this bank. Not ranked here. Checked / reported: 2026-11-02 · Samples: 40 Snapshot: 2026-11-01 · Revision: f5fa9cfafccd97fcfbdb0336e14284faaa8d2d35 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
System 1 Decision Benchmark v2 · P02 · Recognizing multiple requested actions	52.500%	Setup & source Recognizing multiple requested actions: 40 questions. Respan Span-02 respan/span-02-20261015; Hosted native; returned identities {"respan/span-02-20261015": 1934}; collected 2026-10-18 to 2026-10-31 America/Toronto; one primary response per input, no scored retries. Exact aggregate from the release statistics. Development benchmark, not an independently sampled test: 1,934 questions in 41 categories, the frozen 1,342-question bank plus 192 logical and 400 practical additions. The bank was built with model-guided development, and Jev plus retained Microsoft and Drex feedback are development exposure. Related variants and a single recorded primary response per input limit generalization. The scoring key stays private, so public checks cover coverage, timing and aggregate consistency but cannot rescore answers. Strict accuracy counts a wrong usable answer and an unsuccessful response in the same fixed denominator. Timing pools primary client times with the network included; providers, precision, concurrency, routes and dates differ, retained and new timings can span sessions, and quick failures can reduce all-primary latency. The source published a separate cost ledger on 1 November 2026, stored as decision-reference-v2-cost-per-1000, which supersedes its earlier “cost not consolidated” note. Older 1,342-only scores are not converted into this bank. Not ranked here. Checked / reported: 2026-11-02 · Samples: 40 Snapshot: 2026-11-01 · Revision: f5fa9cfafccd97fcfbdb0336e14284faaa8d2d35 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
System 1 Decision Benchmark v2 · P03 · Tracking current conversational intention	55.000%	Setup & source Tracking current conversational intention: 40 questions. Respan Span-02 respan/span-02-20261015; Hosted native; returned identities {"respan/span-02-20261015": 1934}; collected 2026-10-18 to 2026-10-31 America/Toronto; one primary response per input, no scored retries. Exact aggregate from the release statistics. Development benchmark, not an independently sampled test: 1,934 questions in 41 categories, the frozen 1,342-question bank plus 192 logical and 400 practical additions. The bank was built with model-guided development, and Jev plus retained Microsoft and Drex feedback are development exposure. Related variants and a single recorded primary response per input limit generalization. The scoring key stays private, so public checks cover coverage, timing and aggregate consistency but cannot rescore answers. Strict accuracy counts a wrong usable answer and an unsuccessful response in the same fixed denominator. Timing pools primary client times with the network included; providers, precision, concurrency, routes and dates differ, retained and new timings can span sessions, and quick failures can reduce all-primary latency. The source published a separate cost ledger on 1 November 2026, stored as decision-reference-v2-cost-per-1000, which supersedes its earlier “cost not consolidated” note. Older 1,342-only scores are not converted into this bank. Not ranked here. Checked / reported: 2026-11-02 · Samples: 40 Snapshot: 2026-11-01 · Revision: f5fa9cfafccd97fcfbdb0336e14284faaa8d2d35 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
System 1 Decision Benchmark v2 · P04 · Selecting a tool or requesting clarification	47.500%	Setup & source Selecting a tool or requesting clarification: 40 questions. Respan Span-02 respan/span-02-20261015; Hosted native; returned identities {"respan/span-02-20261015": 1934}; collected 2026-10-18 to 2026-10-31 America/Toronto; one primary response per input, no scored retries. Exact aggregate from the release statistics. Development benchmark, not an independently sampled test: 1,934 questions in 41 categories, the frozen 1,342-question bank plus 192 logical and 400 practical additions. The bank was built with model-guided development, and Jev plus retained Microsoft and Drex feedback are development exposure. Related variants and a single recorded primary response per input limit generalization. The scoring key stays private, so public checks cover coverage, timing and aggregate consistency but cannot rescore answers. Strict accuracy counts a wrong usable answer and an unsuccessful response in the same fixed denominator. Timing pools primary client times with the network included; providers, precision, concurrency, routes and dates differ, retained and new timings can span sessions, and quick failures can reduce all-primary latency. The source published a separate cost ledger on 1 November 2026, stored as decision-reference-v2-cost-per-1000, which supersedes its earlier “cost not consolidated” note. Older 1,342-only scores are not converted into this bank. Not ranked here. Checked / reported: 2026-11-02 · Samples: 40 Snapshot: 2026-11-01 · Revision: f5fa9cfafccd97fcfbdb0336e14284faaa8d2d35 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
System 1 Decision Benchmark v2 · P05 · Checking authorization and scope	52.500%	Setup & source Checking authorization and scope: 40 questions. Respan Span-02 respan/span-02-20261015; Hosted native; returned identities {"respan/span-02-20261015": 1934}; collected 2026-10-18 to 2026-10-31 America/Toronto; one primary response per input, no scored retries. Exact aggregate from the release statistics. Development benchmark, not an independently sampled test: 1,934 questions in 41 categories, the frozen 1,342-question bank plus 192 logical and 400 practical additions. The bank was built with model-guided development, and Jev plus retained Microsoft and Drex feedback are development exposure. Related variants and a single recorded primary response per input limit generalization. The scoring key stays private, so public checks cover coverage, timing and aggregate consistency but cannot rescore answers. Strict accuracy counts a wrong usable answer and an unsuccessful response in the same fixed denominator. Timing pools primary client times with the network included; providers, precision, concurrency, routes and dates differ, retained and new timings can span sessions, and quick failures can reduce all-primary latency. The source published a separate cost ledger on 1 November 2026, stored as decision-reference-v2-cost-per-1000, which supersedes its earlier “cost not consolidated” note. Older 1,342-only scores are not converted into this bank. Not ranked here. Checked / reported: 2026-11-02 · Samples: 40 Snapshot: 2026-11-01 · Revision: f5fa9cfafccd97fcfbdb0336e14284faaa8d2d35 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
System 1 Decision Benchmark v2 · P06 · Making decisions with untrusted content	50.000%	Setup & source Making decisions with untrusted content: 40 questions. Respan Span-02 respan/span-02-20261015; Hosted native; returned identities {"respan/span-02-20261015": 1934}; collected 2026-10-18 to 2026-10-31 America/Toronto; one primary response per input, no scored retries. Exact aggregate from the release statistics. Development benchmark, not an independently sampled test: 1,934 questions in 41 categories, the frozen 1,342-question bank plus 192 logical and 400 practical additions. The bank was built with model-guided development, and Jev plus retained Microsoft and Drex feedback are development exposure. Related variants and a single recorded primary response per input limit generalization. The scoring key stays private, so public checks cover coverage, timing and aggregate consistency but cannot rescore answers. Strict accuracy counts a wrong usable answer and an unsuccessful response in the same fixed denominator. Timing pools primary client times with the network included; providers, precision, concurrency, routes and dates differ, retained and new timings can span sessions, and quick failures can reduce all-primary latency. The source published a separate cost ledger on 1 November 2026, stored as decision-reference-v2-cost-per-1000, which supersedes its earlier “cost not consolidated” note. Older 1,342-only scores are not converted into this bank. Not ranked here. Checked / reported: 2026-11-02 · Samples: 40 Snapshot: 2026-11-01 · Revision: f5fa9cfafccd97fcfbdb0336e14284faaa8d2d35 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
System 1 Decision Benchmark v2 · P07 · Applying a contextual content policy	72.500%	Setup & source Applying a contextual content policy: 40 questions. Respan Span-02 respan/span-02-20261015; Hosted native; returned identities {"respan/span-02-20261015": 1934}; collected 2026-10-18 to 2026-10-31 America/Toronto; one primary response per input, no scored retries. Exact aggregate from the release statistics. Development benchmark, not an independently sampled test: 1,934 questions in 41 categories, the frozen 1,342-question bank plus 192 logical and 400 practical additions. The bank was built with model-guided development, and Jev plus retained Microsoft and Drex feedback are development exposure. Related variants and a single recorded primary response per input limit generalization. The scoring key stays private, so public checks cover coverage, timing and aggregate consistency but cannot rescore answers. Strict accuracy counts a wrong usable answer and an unsuccessful response in the same fixed denominator. Timing pools primary client times with the network included; providers, precision, concurrency, routes and dates differ, retained and new timings can span sessions, and quick failures can reduce all-primary latency. The source published a separate cost ledger on 1 November 2026, stored as decision-reference-v2-cost-per-1000, which supersedes its earlier “cost not consolidated” note. Older 1,342-only scores are not converted into this bank. Not ranked here. Checked / reported: 2026-11-02 · Samples: 40 Snapshot: 2026-11-01 · Revision: f5fa9cfafccd97fcfbdb0336e14284faaa8d2d35 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
System 1 Decision Benchmark v2 · P08 · Extracting and normalizing facts	32.500%	Setup & source Extracting and normalizing facts: 40 questions. Respan Span-02 respan/span-02-20261015; Hosted native; returned identities {"respan/span-02-20261015": 1934}; collected 2026-10-18 to 2026-10-31 America/Toronto; one primary response per input, no scored retries. Exact aggregate from the release statistics. Development benchmark, not an independently sampled test: 1,934 questions in 41 categories, the frozen 1,342-question bank plus 192 logical and 400 practical additions. The bank was built with model-guided development, and Jev plus retained Microsoft and Drex feedback are development exposure. Related variants and a single recorded primary response per input limit generalization. The scoring key stays private, so public checks cover coverage, timing and aggregate consistency but cannot rescore answers. Strict accuracy counts a wrong usable answer and an unsuccessful response in the same fixed denominator. Timing pools primary client times with the network included; providers, precision, concurrency, routes and dates differ, retained and new timings can span sessions, and quick failures can reduce all-primary latency. The source published a separate cost ledger on 1 November 2026, stored as decision-reference-v2-cost-per-1000, which supersedes its earlier “cost not consolidated” note. Older 1,342-only scores are not converted into this bank. Not ranked here. Checked / reported: 2026-11-02 · Samples: 40 Snapshot: 2026-11-01 · Revision: f5fa9cfafccd97fcfbdb0336e14284faaa8d2d35 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
System 1 Decision Benchmark v2 · P09 · Checking claims against supplied evidence	47.500%	Setup & source Checking claims against supplied evidence: 40 questions. Respan Span-02 respan/span-02-20261015; Hosted native; returned identities {"respan/span-02-20261015": 1934}; collected 2026-10-18 to 2026-10-31 America/Toronto; one primary response per input, no scored retries. Exact aggregate from the release statistics. Development benchmark, not an independently sampled test: 1,934 questions in 41 categories, the frozen 1,342-question bank plus 192 logical and 400 practical additions. The bank was built with model-guided development, and Jev plus retained Microsoft and Drex feedback are development exposure. Related variants and a single recorded primary response per input limit generalization. The scoring key stays private, so public checks cover coverage, timing and aggregate consistency but cannot rescore answers. Strict accuracy counts a wrong usable answer and an unsuccessful response in the same fixed denominator. Timing pools primary client times with the network included; providers, precision, concurrency, routes and dates differ, retained and new timings can span sessions, and quick failures can reduce all-primary latency. The source published a separate cost ledger on 1 November 2026, stored as decision-reference-v2-cost-per-1000, which supersedes its earlier “cost not consolidated” note. Older 1,342-only scores are not converted into this bank. Not ranked here. Checked / reported: 2026-11-02 · Samples: 40 Snapshot: 2026-11-01 · Revision: f5fa9cfafccd97fcfbdb0336e14284faaa8d2d35 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
System 1 Decision Benchmark v2 · P10 · Assigning severity or priority from a rubric	35.000%	Setup & source Assigning severity or priority from a rubric: 40 questions. Respan Span-02 respan/span-02-20261015; Hosted native; returned identities {"respan/span-02-20261015": 1934}; collected 2026-10-18 to 2026-10-31 America/Toronto; one primary response per input, no scored retries. Exact aggregate from the release statistics. Development benchmark, not an independently sampled test: 1,934 questions in 41 categories, the frozen 1,342-question bank plus 192 logical and 400 practical additions. The bank was built with model-guided development, and Jev plus retained Microsoft and Drex feedback are development exposure. Related variants and a single recorded primary response per input limit generalization. The scoring key stays private, so public checks cover coverage, timing and aggregate consistency but cannot rescore answers. Strict accuracy counts a wrong usable answer and an unsuccessful response in the same fixed denominator. Timing pools primary client times with the network included; providers, precision, concurrency, routes and dates differ, retained and new timings can span sessions, and quick failures can reduce all-primary latency. The source published a separate cost ledger on 1 November 2026, stored as decision-reference-v2-cost-per-1000, which supersedes its earlier “cost not consolidated” note. Older 1,342-only scores are not converted into this bank. Not ranked here. Checked / reported: 2026-11-02 · Samples: 40 Snapshot: 2026-11-01 · Revision: f5fa9cfafccd97fcfbdb0336e14284faaa8d2d35 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
System 1 Decision Benchmark v2 · Reported cost per 1,000 questions	0.00532 USD/1k decisions	Setup & source Provider-reported charges for the primary calls of each system on the v2 bank, in USD per 1,000 questions. Respan Span-02 (20261015); System 1 v2 route; evidence reported. Provider-reported charges for all 1,934 selected calls. Costs come from the selected primary calls on the same 1,934 inputs as the scores. Warmups, probes, recovery attempts, model reviews and account balances are excluded. The value is the known run subtotal divided by 1,934 and multiplied by 1,000, not the bill for a separate 1,000-call run. A partial value is only the known subtotal: missing charges stay missing, so the true cost is higher by an unknown amount. Providers, routes, dates and access terms differ, and these are not current tariffs. D3 Ultra used free public access without a metered receipt and has no row. Development benchmark; not ranked here. Checked / reported: 2026-11-02 · Samples: 1934 Snapshot: 2026-11-02 · Revision: 5b47806ba527db5e1abc88578789f3eae6e8e646 · Published summary observations; upstream task/result terms retained Result source · Benchmark source

Independent · ModelTested

Span-02 · Independent · ModelTested
Evaluation	Result	Evidence
ModelTested 56-case · All-case accuracy	96.40%	Setup & source Exact named hosted API observation; All-case accuracy. Named source route respan/span-02; ModelTested runner v1; tested_on=openrouter. Backend weight revision and API inference precision unpublished. Publisher generated 2026-11-01; board added this route 2026-10-16. Execution dates, token usage and hashes remain in its per-run receipts. No private prompt/response archive imported. 56 authored situations, including 19 hard cases, each with two typed questions; 14 public and 42 hidden cases. Source aggregate averages per-case decision accuracy. Confidence score is 100×(1−mean multiclass Brier/2), not ECE or a calibration probability. Small single-run benchmark; current HF access/size does not inherit this evaluator’s stale closed/API-only metadata. No tariff inferred from rounded cost 0.0. Rows added 1 November 2026 use revision 668a52ad3b932ca910fd985f96dcdf0a85d2dae1, where the earlier 11 systems are unchanged. Rows added 2 November 2026 use revision 5c6c19bb3b7332ccd28c8c13bd32cdc69cedc8f0. Checked / reported: 2026-11-01 · Samples: 56 Snapshot: 2026-10-18 · Revision: 081e8f37130a4a9077b8cffe62827d4c3b9e3b23; 668a52ad3b932ca910fd985f96dcdf0a85d2dae1 for Drex Max, Jev 1.14 and d1.1; 5c6c19bb3b7332ccd28c8c13bd32cdc69cedc8f0 for Mammoth, Elephant, Joey, Span-02 and both Solar Decide routes · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
ModelTested 56-case · Hard-case accuracy	94.70%	Setup & source Exact named hosted API observation; Hard-case accuracy. Named source route respan/span-02; ModelTested runner v1; tested_on=openrouter. Backend weight revision and API inference precision unpublished. Publisher generated 2026-11-01; board added this route 2026-10-16. Execution dates, token usage and hashes remain in its per-run receipts. No private prompt/response archive imported. 56 authored situations, including 19 hard cases, each with two typed questions; 14 public and 42 hidden cases. Source aggregate averages per-case decision accuracy. Confidence score is 100×(1−mean multiclass Brier/2), not ECE or a calibration probability. Small single-run benchmark; current HF access/size does not inherit this evaluator’s stale closed/API-only metadata. No tariff inferred from rounded cost 0.0. Rows added 1 November 2026 use revision 668a52ad3b932ca910fd985f96dcdf0a85d2dae1, where the earlier 11 systems are unchanged. Rows added 2 November 2026 use revision 5c6c19bb3b7332ccd28c8c13bd32cdc69cedc8f0. Checked / reported: 2026-11-01 · Samples: 56 Snapshot: 2026-10-18 · Revision: 081e8f37130a4a9077b8cffe62827d4c3b9e3b23; 668a52ad3b932ca910fd985f96dcdf0a85d2dae1 for Drex Max, Jev 1.14 and d1.1; 5c6c19bb3b7332ccd28c8c13bd32cdc69cedc8f0 for Mammoth, Elephant, Joey, Span-02 and both Solar Decide routes · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
ModelTested 56-case · Public-case accuracy	96.40%	Setup & source Exact named hosted API observation; Public-case accuracy. Named source route respan/span-02; ModelTested runner v1; tested_on=openrouter. Backend weight revision and API inference precision unpublished. Publisher generated 2026-11-01; board added this route 2026-10-16. Execution dates, token usage and hashes remain in its per-run receipts. No private prompt/response archive imported. 56 authored situations, including 19 hard cases, each with two typed questions; 14 public and 42 hidden cases. Source aggregate averages per-case decision accuracy. Confidence score is 100×(1−mean multiclass Brier/2), not ECE or a calibration probability. Small single-run benchmark; current HF access/size does not inherit this evaluator’s stale closed/API-only metadata. No tariff inferred from rounded cost 0.0. Rows added 1 November 2026 use revision 668a52ad3b932ca910fd985f96dcdf0a85d2dae1, where the earlier 11 systems are unchanged. Rows added 2 November 2026 use revision 5c6c19bb3b7332ccd28c8c13bd32cdc69cedc8f0. Checked / reported: 2026-11-01 · Samples: 56 Snapshot: 2026-10-18 · Revision: 081e8f37130a4a9077b8cffe62827d4c3b9e3b23; 668a52ad3b932ca910fd985f96dcdf0a85d2dae1 for Drex Max, Jev 1.14 and d1.1; 5c6c19bb3b7332ccd28c8c13bd32cdc69cedc8f0 for Mammoth, Elephant, Joey, Span-02 and both Solar Decide routes · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
ModelTested 56-case · Hidden-case accuracy	96.40%	Setup & source Exact named hosted API observation; Hidden-case accuracy. Named source route respan/span-02; ModelTested runner v1; tested_on=openrouter. Backend weight revision and API inference precision unpublished. Publisher generated 2026-11-01; board added this route 2026-10-16. Execution dates, token usage and hashes remain in its per-run receipts. No private prompt/response archive imported. 56 authored situations, including 19 hard cases, each with two typed questions; 14 public and 42 hidden cases. Source aggregate averages per-case decision accuracy. Confidence score is 100×(1−mean multiclass Brier/2), not ECE or a calibration probability. Small single-run benchmark; current HF access/size does not inherit this evaluator’s stale closed/API-only metadata. No tariff inferred from rounded cost 0.0. Rows added 1 November 2026 use revision 668a52ad3b932ca910fd985f96dcdf0a85d2dae1, where the earlier 11 systems are unchanged. Rows added 2 November 2026 use revision 5c6c19bb3b7332ccd28c8c13bd32cdc69cedc8f0. Checked / reported: 2026-11-01 · Samples: 56 Snapshot: 2026-10-18 · Revision: 081e8f37130a4a9077b8cffe62827d4c3b9e3b23; 668a52ad3b932ca910fd985f96dcdf0a85d2dae1 for Drex Max, Jev 1.14 and d1.1; 5c6c19bb3b7332ccd28c8c13bd32cdc69cedc8f0 for Mammoth, Elephant, Joey, Span-02 and both Solar Decide routes · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
ModelTested 56-case · Brier-based confidence score	96.40	Setup & source Exact named hosted API observation; Brier-based confidence score. Named source route respan/span-02; ModelTested runner v1; tested_on=openrouter. Backend weight revision and API inference precision unpublished. Publisher generated 2026-11-01; board added this route 2026-10-16. Execution dates, token usage and hashes remain in its per-run receipts. No private prompt/response archive imported. 56 authored situations, including 19 hard cases, each with two typed questions; 14 public and 42 hidden cases. Source aggregate averages per-case decision accuracy. Confidence score is 100×(1−mean multiclass Brier/2), not ECE or a calibration probability. Small single-run benchmark; current HF access/size does not inherit this evaluator’s stale closed/API-only metadata. No tariff inferred from rounded cost 0.0. Rows added 1 November 2026 use revision 668a52ad3b932ca910fd985f96dcdf0a85d2dae1, where the earlier 11 systems are unchanged. Rows added 2 November 2026 use revision 5c6c19bb3b7332ccd28c8c13bd32cdc69cedc8f0. Checked / reported: 2026-11-01 · Samples: 56 Snapshot: 2026-10-18 · Revision: 081e8f37130a4a9077b8cffe62827d4c3b9e3b23; 668a52ad3b932ca910fd985f96dcdf0a85d2dae1 for Drex Max, Jev 1.14 and d1.1; 5c6c19bb3b7332ccd28c8c13bd32cdc69cedc8f0 for Mammoth, Elephant, Joey, Span-02 and both Solar Decide routes · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source

Evaluator-published · S1MB (community measured runs)

Span-02 · Evaluator-published · S1MB (community measured runs)
Evaluation	Result	Evidence
S1MB · aegis2-noul-test-v1 · binary_brier	0.2034 Brier	Setup & source S1MB datasets/aegis2 / test · binary_brier. Population 100 cases / 148 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 100, "expected": 148, "succeeded": 148, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 100 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · aqua-rat-choice-test-v1 · target_mass_at_prediction	82.00%	Setup & source S1MB datasets/aqua_rat / test · target_mass_at_prediction. Population 100 cases / 100 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 100 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · arc-choice-test-v1 · target_mass_at_prediction	98.00%	Setup & source S1MB datasets/arc / test · target_mass_at_prediction. Population 100 cases / 100 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 100 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · arct-choice-test-v1 · target_mass_at_prediction	86.00%	Setup & source S1MB datasets/arct / test · target_mass_at_prediction. Population 100 cases / 100 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 100 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · argument-quality-choice-test-v1 · target_mass_at_prediction	91.00%	Setup & source S1MB datasets/argument_quality / test · target_mass_at_prediction. Population 100 cases / 100 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 100 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · argument-quality-noul-test-v1 · binary_brier	0.1185 Brier	Setup & source S1MB datasets/argument_quality / test · binary_brier. Population 100 cases / 100 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 100 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · babi-nli-noul-test-v1 · binary_brier	0.1126 Brier	Setup & source S1MB datasets/babi_nli / test · binary_brier. Population 100 cases / 100 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 100 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · banking77-choice-test-v1 · target_mass_at_prediction	77.00%	Setup & source S1MB datasets/banking77 / test · target_mass_at_prediction. Population 100 cases / 100 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 100 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · bbq-choice-test-v1 · target_mass_at_prediction	96.00%	Setup & source S1MB datasets/bbq / test · target_mass_at_prediction. Population 100 cases / 100 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 100 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · boardgameqa-choice-test-v1 · target_mass_at_prediction	56.00%	Setup & source S1MB datasets/boardgameqa / test · target_mass_at_prediction. Population 100 cases / 100 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 100 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · canttalk-noul-test-v1 · binary_brier	0.2071 Brier	Setup & source S1MB datasets/canttalk / test · binary_brier. Population 100 cases / 100 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 100 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · civil-comments-noul-test-v1 · binary_brier	0.1010 Brier	Setup & source S1MB datasets/civil_comments / test · binary_brier. Population 100 cases / 700 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 100, "expected": 700, "succeeded": 700, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 100 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · cladder-noul-test-v1 · binary_brier	0.1620 Brier	Setup & source S1MB datasets/cladder / test · binary_brier. Population 100 cases / 100 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 100 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · clinc-choice-test-v1 · target_mass_at_prediction	91.39%	Setup & source S1MB datasets/clinc / test · target_mass_at_prediction. Population 151 cases / 151 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 151, "expected": 151, "succeeded": 151, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 151 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · commonsensemoralja-noul · binary_brier	0.0693 Brier	Setup & source S1MB datasets/commonsensemoralja-noul / test · binary_brier. Population 300 cases / 300 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 300, "expected": 300, "succeeded": 300, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 300 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · contract-nli-choice-test-v1 · target_mass_at_prediction	80.18%	Setup & source S1MB datasets/contract_nli / test · target_mass_at_prediction. Population 100 cases / 1700 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 100, "expected": 1700, "succeeded": 1700, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 100 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · corr2cause-choice-test-v1 · target_mass_at_prediction	42.00%	Setup & source S1MB datasets/corr2cause / test · target_mass_at_prediction. Population 100 cases / 100 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 100 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · creak-noul-test-v1 · binary_brier	0.0740 Brier	Setup & source S1MB datasets/creak / test · binary_brier. Population 100 cases / 100 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 100 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · crows-pairs-choice-test-v1 · target_mass_at_prediction	86.00%	Setup & source S1MB datasets/crows_pairs / test · target_mass_at_prediction. Population 100 cases / 100 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 100 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · cvbench-depth-choice · target_mass_at_prediction	88.00%	Setup & source S1MB datasets/cvbench-depth-choice / test · target_mass_at_prediction. Population 300 cases / 300 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 300, "expected": 300, "succeeded": 300, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 300 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · cvbench-distance-choice · target_mass_at_prediction	86.67%	Setup & source S1MB datasets/cvbench-distance-choice / test · target_mass_at_prediction. Population 300 cases / 300 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 300, "expected": 300, "succeeded": 300, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 300 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · cvbench-relation-choice · target_mass_at_prediction	96.00%	Setup & source S1MB datasets/cvbench-relation-choice / test · target_mass_at_prediction. Population 300 cases / 300 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 300, "expected": 300, "succeeded": 300, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 300 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · dbpedia-choice-test-v1 · target_mass_at_prediction	97.00%	Setup & source S1MB datasets/dbpedia / test · target_mass_at_prediction. Population 100 cases / 100 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 100 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · defeasible-nli-choice-test-v1 · target_mass_at_prediction	89.00%	Setup & source S1MB datasets/defeasible_nli / test · target_mass_at_prediction. Population 100 cases / 100 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 100 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · esci-choice-test-v1 · target_mass_at_prediction	57.00%	Setup & source S1MB datasets/esci / test · target_mass_at_prediction. Population 100 cases / 100 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 100 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · ethics-choice-test-v1 · target_mass_at_prediction	94.44%	Setup & source S1MB datasets/ethics / test · target_mass_at_prediction. Population 18 cases / 18 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 18, "expected": 18, "succeeded": 18, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 18 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · ethics-noul-test-v1 · binary_brier	0.1491 Brier	Setup & source S1MB datasets/ethics / test · binary_brier. Population 82 cases / 82 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 82, "expected": 82, "succeeded": 82, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 82 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · ethos-noul-test-v1 · binary_brier	0.1562 Brier	Setup & source S1MB datasets/ethos / test · binary_brier. Population 100 cases / 100 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 100 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · few-nerd-choice-test-v1 · target_mass_at_prediction	81.01%	Setup & source S1MB datasets/few_nerd / test · target_mass_at_prediction. Population 100 cases / 358 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 100, "expected": 358, "succeeded": 358, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 100 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · fol-nli-choice-test-v1 · target_mass_at_prediction	64.00%	Setup & source S1MB datasets/fol_nli / test · target_mass_at_prediction. Population 100 cases / 100 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 100 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · followir-core17-noul-test-v1 · binary_brier	0.1727 Brier	Setup & source S1MB datasets/followir_core17 / test · binary_brier. Population 100 cases / 200 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 100, "expected": 200, "succeeded": 200, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 100 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · followir-news21-noul-test-v1 · binary_brier	0.1889 Brier	Setup & source S1MB datasets/followir_news21 / test · binary_brier. Population 100 cases / 200 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 100, "expected": 200, "succeeded": 200, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 100 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · followir-robust04-noul-test-v1 · binary_brier	0.0409 Brier	Setup & source S1MB datasets/followir_robust04 / test · binary_brier. Population 100 cases / 200 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 100, "expected": 200, "succeeded": 200, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 100 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · generalization-japanese-diverse-choice · target_mass_at_prediction	99.67%	Setup & source S1MB datasets/generalization-japanese-diverse-choice / test · target_mass_at_prediction. Population 300 cases / 300 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 300, "expected": 300, "succeeded": 300, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 300 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · generalization-japanese-diverse-noul · binary_brier	0.0076 Brier	Setup & source S1MB datasets/generalization-japanese-diverse-noul / test · binary_brier. Population 300 cases / 300 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 300, "expected": 300, "succeeded": 300, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 300 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · generalization-japanese-knowledge-hard-choice · target_mass_at_prediction	99.67%	Setup & source S1MB datasets/generalization-japanese-knowledge-hard-choice / test · target_mass_at_prediction. Population 300 cases / 300 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 300, "expected": 300, "succeeded": 300, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 300 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · generalization-japanese-knowledge-hard-noul · binary_brier	0.0826 Brier	Setup & source S1MB datasets/generalization-japanese-knowledge-hard-noul / test · binary_brier. Population 300 cases / 300 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 300, "expected": 300, "succeeded": 300, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 300 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · generalization-japanese-knowledge-hard-score · normalized_expected_score_mae	0.0682 MAE	Setup & source S1MB datasets/generalization-japanese-knowledge-hard-score / test · normalized_expected_score_mae. Population 300 cases / 300 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 300, "expected": 300, "succeeded": 300, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 300 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · generalization-japanese-math-formal-hard-choice · target_mass_at_prediction	56.67%	Setup & source S1MB datasets/generalization-japanese-math-formal-hard-choice / test · target_mass_at_prediction. Population 300 cases / 300 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 300, "expected": 300, "succeeded": 300, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 300 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · generalization-japanese-math-formal-hard-noul · binary_brier	0.2794 Brier	Setup & source S1MB datasets/generalization-japanese-math-formal-hard-noul / test · binary_brier. Population 300 cases / 300 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 300, "expected": 300, "succeeded": 300, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 300 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · generalization-japanese-math-formal-hard-score · normalized_expected_score_mae	0.2985 MAE	Setup & source S1MB datasets/generalization-japanese-math-formal-hard-score / test · normalized_expected_score_mae. Population 300 cases / 300 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 300, "expected": 300, "succeeded": 300, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 300 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · generalization-vision-easy-choice · target_mass_at_prediction	97.67%	Setup & source S1MB datasets/generalization-vision-easy-choice / test · target_mass_at_prediction. Population 300 cases / 300 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 300, "expected": 300, "succeeded": 300, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 300 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · generalization-vision-easy-noul · binary_brier	0.0147 Brier	Setup & source S1MB datasets/generalization-vision-easy-noul / test · binary_brier. Population 300 cases / 300 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 300, "expected": 300, "succeeded": 300, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 300 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · generalization-vision-easy-score · normalized_expected_score_mae	0.0836 MAE	Setup & source S1MB datasets/generalization-vision-easy-score / test · normalized_expected_score_mae. Population 300 cases / 300 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 300, "expected": 300, "succeeded": 300, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 300 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · generalization-vision-hard-choice · target_mass_at_prediction	75.33%	Setup & source S1MB datasets/generalization-vision-hard-choice / test · target_mass_at_prediction. Population 300 cases / 300 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 300, "expected": 300, "succeeded": 300, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 300 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · generalization-vision-hard-noul · binary_brier	0.1532 Brier	Setup & source S1MB datasets/generalization-vision-hard-noul / test · binary_brier. Population 300 cases / 300 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 300, "expected": 300, "succeeded": 300, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 300 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · generalization-vision-hard-score · normalized_expected_score_mae	0.1780 MAE	Setup & source S1MB datasets/generalization-vision-hard-score / test · normalized_expected_score_mae. Population 300 cases / 300 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 300, "expected": 300, "succeeded": 300, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 300 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · go-emotions-noul-test-v1 · binary_brier	0.0981 Brier	Setup & source S1MB datasets/go_emotions / test · binary_brier. Population 100 cases / 2800 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 100, "expected": 2800, "succeeded": 2800, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 100 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · gpqa-ja-main-choice · target_mass_at_prediction	49.67%	Setup & source S1MB datasets/gpqa-ja-main-choice / test · target_mass_at_prediction. Population 300 cases / 300 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 300, "expected": 300, "succeeded": 300, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 300 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · gretel-pii-choice-test-v1 · target_mass_at_prediction	85.55%	Setup & source S1MB datasets/gretel_pii / test · target_mass_at_prediction. Population 100 cases / 353 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 100, "expected": 353, "succeeded": 353, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 100 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · gsm8k-choice-test-v1 · target_mass_at_prediction	95.00%	Setup & source S1MB datasets/gsm8k / test · target_mass_at_prediction. Population 100 cases / 100 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 100 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · hans-noul-test-v1 · binary_brier	0.0721 Brier	Setup & source S1MB datasets/hans / test · binary_brier. Population 100 cases / 100 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 100 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · hatecheck-noul-test-v1 · binary_brier	0.0051 Brier	Setup & source S1MB datasets/hatecheck / test · binary_brier. Population 100 cases / 100 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 100 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · hh-rlhf-choice-test-v1 · target_mass_at_prediction	68.00%	Setup & source S1MB datasets/hh_rlhf / test · target_mass_at_prediction. Population 100 cases / 100 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 100 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · hwu64-choice-test-v1 · target_mass_at_prediction	84.00%	Setup & source S1MB datasets/hwu64 / test · target_mass_at_prediction. Population 100 cases / 100 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 100 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · impli-noul-test-v1 · binary_brier	0.0619 Brier	Setup & source S1MB datasets/impli / test · binary_brier. Population 100 cases / 100 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 100 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · jamc-qa-choice · target_mass_at_prediction	62.33%	Setup & source S1MB datasets/jamc-qa-choice / test · target_mass_at_prediction. Population 300 cases / 300 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 300, "expected": 300, "succeeded": 300, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 300 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · jamp-choice · target_mass_at_prediction	80.33%	Setup & source S1MB datasets/jamp-choice / test · target_mass_at_prediction. Population 300 cases / 300 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 300, "expected": 300, "succeeded": 300, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 300 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · janli-choice · target_mass_at_prediction	93.68%	Setup & source S1MB datasets/janli-choice / test · target_mass_at_prediction. Population 285 cases / 285 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 285, "expected": 285, "succeeded": 285, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 285 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · jcola-noul · binary_brier	0.2378 Brier	Setup & source S1MB datasets/jcola-noul / test · binary_brier. Population 299 cases / 299 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 299, "expected": 299, "succeeded": 299, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 299 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · jcommonsenseqa-choice · target_mass_at_prediction	97.00%	Setup & source S1MB datasets/jcommonsenseqa-choice / test · target_mass_at_prediction. Population 300 cases / 300 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 300, "expected": 300, "succeeded": 300, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 300 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · jculture-mcq-noul · binary_brier	0.2819 Brier	Setup & source S1MB datasets/jculture-mcq-noul / test · binary_brier. Population 283 cases / 283 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 283, "expected": 283, "succeeded": 283, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 283 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · jmmlu-japan-choice · target_mass_at_prediction	91.33%	Setup & source S1MB datasets/jmmlu-japan-choice / test · target_mass_at_prediction. Population 300 cases / 300 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 300, "expected": 300, "succeeded": 300, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 300 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · jmteb-retrieval · target_mass_at_prediction	96.33%	Setup & source S1MB datasets/jmteb-retrieval / test · target_mass_at_prediction. Population 300 cases / 300 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 300, "expected": 300, "succeeded": 300, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 300 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · jnli-choice · target_mass_at_prediction	76.67%	Setup & source S1MB datasets/jnli-choice / test · target_mass_at_prediction. Population 300 cases / 300 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 300, "expected": 300, "succeeded": 300, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 300 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · jsem-choice · target_mass_at_prediction	77.00%	Setup & source S1MB datasets/jsem-choice / test · target_mass_at_prediction. Population 300 cases / 300 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 300, "expected": 300, "succeeded": 300, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 300 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · jsick-sts-score · normalized_expected_score_mae	0.1826 MAE	Setup & source S1MB datasets/jsick-sts-score / test · normalized_expected_score_mae. Population 300 cases / 300 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 300, "expected": 300, "succeeded": 300, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 300 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · jsts-score · normalized_expected_score_mae	0.0940 MAE	Setup & source S1MB datasets/jsts-score / test · normalized_expected_score_mae. Population 300 cases / 300 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 300, "expected": 300, "succeeded": 300, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 300 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · koniq-score · normalized_expected_score_mae	0.1780 MAE	Setup & source S1MB datasets/koniq-score / test · normalized_expected_score_mae. Population 300 cases / 300 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 300, "expected": 300, "succeeded": 300, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 300 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · kuci-choice · target_mass_at_prediction	81.67%	Setup & source S1MB datasets/kuci-choice / test · target_mass_at_prediction. Population 300 cases / 300 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 300, "expected": 300, "succeeded": 300, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 300 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · laya--ag-news-choice-test-v1 · target_mass_at_prediction	86.00%	Setup & source S1MB datasets/laya__ag_news / test · target_mass_at_prediction. Population 100 cases / 100 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 100 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · laya--enron-spam-noul-test-v1 · binary_brier	0.0373 Brier	Setup & source S1MB datasets/laya__enron_spam / test · binary_brier. Population 100 cases / 100 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 100 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · laya--typed-decisions-choice-test-v1 · target_mass_at_prediction	46.85%	Setup & source S1MB datasets/laya__typed_decisions / test · target_mass_at_prediction. Population 100 cases / 148 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 100, "expected": 148, "succeeded": 148, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 100 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · laya--typed-decisions-noul-test-v1 · binary_brier	0.0876 Brier	Setup & source S1MB datasets/laya__typed_decisions / test · binary_brier. Population 100 cases / 152 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 100, "expected": 152, "succeeded": 152, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 100 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · laya--typed-decisions-score-test-v1 · normalized_expected_score_mae	0.1046 MAE	Setup & source S1MB datasets/laya__typed_decisions / test · normalized_expected_score_mae. Population 100 cases / 200 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 100, "expected": 200, "succeeded": 200, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 100 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · lexcomp-noul-test-v1 · binary_brier	0.2096 Brier	Setup & source S1MB datasets/lexcomp / test · binary_brier. Population 100 cases / 100 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 100 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · logical-entailment-noul-test-v1 · binary_brier	0.2010 Brier	Setup & source S1MB datasets/logical_entailment / test · binary_brier. Population 100 cases / 100 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 100 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · lonli-choice-test-v1 · target_mass_at_prediction	97.00%	Setup & source S1MB datasets/lonli / test · target_mass_at_prediction. Population 100 cases / 100 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 100 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · massive-choice-test-v1 · target_mass_at_prediction	91.00%	Setup & source S1MB datasets/massive / test · target_mass_at_prediction. Population 100 cases / 200 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 100, "expected": 200, "succeeded": 200, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 100 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · mintaka-ja-boolean-noul · binary_brier	0.1382 Brier	Setup & source S1MB datasets/mintaka-ja-boolean-noul / test · binary_brier. Population 300 cases / 300 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 300, "expected": 300, "succeeded": 300, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 300 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · miqa-choice-test-v1 · target_mass_at_prediction	97.00%	Setup & source S1MB datasets/miqa / test · target_mass_at_prediction. Population 100 cases / 100 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 100 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · mmlu-prox-ja-choice · target_mass_at_prediction	66.67%	Setup & source S1MB datasets/mmlu-prox-ja-choice / test · target_mass_at_prediction. Population 300 cases / 300 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 300, "expected": 300, "succeeded": 300, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 300 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · mmmlu-choice · target_mass_at_prediction	80.67%	Setup & source S1MB datasets/mmmlu-choice / test · target_mass_at_prediction. Population 300 cases / 300 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 300, "expected": 300, "succeeded": 300, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 300 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · mtop-choice-test-v1 · target_mass_at_prediction	84.00%	Setup & source S1MB datasets/mtop / test · target_mass_at_prediction. Population 100 cases / 200 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 100, "expected": 200, "succeeded": 200, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 100 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · mvtec-ad-noul · binary_brier	0.3320 Brier	Setup & source S1MB datasets/mvtec-ad-noul / test · binary_brier. Population 300 cases / 300 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 300, "expected": 300, "succeeded": 300, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 300 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · nights-choice · target_mass_at_prediction	84.33%	Setup & source S1MB datasets/nights-choice / test · target_mass_at_prediction. Population 300 cases / 300 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 300, "expected": 300, "succeeded": 300, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 300 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · nlsat-noul-test-v1 · binary_brier	0.3838 Brier	Setup & source S1MB datasets/nlsat / test · binary_brier. Population 100 cases / 100 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 100 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · nlvr2-noul · binary_brier	0.0833 Brier	Setup & source S1MB datasets/nlvr2-noul / test · binary_brier. Population 300 cases / 300 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 300, "expected": 300, "succeeded": 300, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 300 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · open-jev--amount-extraction-control-v1-choice-test-v1 · target_mass_at_prediction	88.89%	Setup & source S1MB datasets/open_jev__amount-extraction-control-v1 / test · target_mass_at_prediction. Population 36 cases / 36 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 36, "expected": 36, "succeeded": 36, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 36 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · open-jev--amount-extraction-control-v1-noul-test-v1 · binary_brier	0.1832 Brier	Setup & source S1MB datasets/open_jev__amount-extraction-control-v1 / test · binary_brier. Population 64 cases / 64 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 64, "expected": 64, "succeeded": 64, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 64 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · open-jev--browser-control-v1-choice-test-v1 · target_mass_at_prediction	100.00%	Setup & source S1MB datasets/open_jev__browser-control-v1 / test · target_mass_at_prediction. Population 100 cases / 100 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 100 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · open-jev--citation-control-v1-choice-test-v1 · target_mass_at_prediction	100.00%	Setup & source S1MB datasets/open_jev__citation-control-v1 / test · target_mass_at_prediction. Population 100 cases / 100 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 100 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · open-jev--context-retention-control-v1-noul-test-v1 · binary_brier	0.0172 Brier	Setup & source S1MB datasets/open_jev__context-retention-control-v1 / test · binary_brier. Population 100 cases / 100 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 100 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · open-jev--customer-control-v1-choice-test-v1 · target_mass_at_prediction	88.24%	Setup & source S1MB datasets/open_jev__customer-control-v1 / test · target_mass_at_prediction. Population 17 cases / 17 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 17, "expected": 17, "succeeded": 17, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 17 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · open-jev--customer-control-v1-noul-test-v1 · binary_brier	0.0007 Brier	Setup & source S1MB datasets/open_jev__customer-control-v1 / test · binary_brier. Population 33 cases / 33 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 33, "expected": 33, "succeeded": 33, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 33 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · open-jev--customer-control-v1-score-test-v1 · normalized_expected_score_mae	0.0719 MAE	Setup & source S1MB datasets/open_jev__customer-control-v1 / test · normalized_expected_score_mae. Population 50 cases / 50 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 50, "expected": 50, "succeeded": 50, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 50 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · open-jev--drone-control-v1-choice-test-v1 · target_mass_at_prediction	18.75%	Setup & source S1MB datasets/open_jev__drone-control-v1 / test · target_mass_at_prediction. Population 32 cases / 32 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 32, "expected": 32, "succeeded": 32, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 32 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · open-jev--drone-control-v1-noul-test-v1 · binary_brier	0.0182 Brier	Setup & source S1MB datasets/open_jev__drone-control-v1 / test · binary_brier. Population 34 cases / 34 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 34, "expected": 34, "succeeded": 34, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 34 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · open-jev--drone-control-v1-score-test-v1 · normalized_expected_score_mae	0.2858 MAE	Setup & source S1MB datasets/open_jev__drone-control-v1 / test · normalized_expected_score_mae. Population 34 cases / 34 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 34, "expected": 34, "succeeded": 34, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 34 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · open-jev--email-selection-control-v1-choice-test-v1 · target_mass_at_prediction	95.83%	Setup & source S1MB datasets/open_jev__email-selection-control-v1 / test · target_mass_at_prediction. Population 48 cases / 48 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 48, "expected": 48, "succeeded": 48, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 48 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · open-jev--email-selection-control-v1-noul-test-v1 · binary_brier	0.0922 Brier	Setup & source S1MB datasets/open_jev__email-selection-control-v1 / test · binary_brier. Population 52 cases / 52 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 52, "expected": 52, "succeeded": 52, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 52 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · open-jev--entity-alignment-control-v1-noul-test-v1 · binary_brier	0.0500 Brier	Setup & source S1MB datasets/open_jev__entity-alignment-control-v1 / test · binary_brier. Population 75 cases / 75 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 75, "expected": 75, "succeeded": 75, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 75 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · open-jev--entity-alignment-control-v1-score-test-v1 · normalized_expected_score_mae	0.1445 MAE	Setup & source S1MB datasets/open_jev__entity-alignment-control-v1 / test · normalized_expected_score_mae. Population 25 cases / 25 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 25, "expected": 25, "succeeded": 25, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 25 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · open-jev--ir-control-v1-choice-test-v1 · target_mass_at_prediction	91.18%	Setup & source S1MB datasets/open_jev__ir-control-v1 / test · target_mass_at_prediction. Population 17 cases / 17 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 17, "expected": 17, "succeeded": 17, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 17 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · open-jev--ir-control-v1-noul-test-v1 · binary_brier	0.0275 Brier	Setup & source S1MB datasets/open_jev__ir-control-v1 / test · binary_brier. Population 28 cases / 28 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 28, "expected": 28, "succeeded": 28, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 28 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · open-jev--ir-control-v1-score-test-v1 · normalized_expected_score_mae	0.0160 MAE	Setup & source S1MB datasets/open_jev__ir-control-v1 / test · normalized_expected_score_mae. Population 55 cases / 55 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 55, "expected": 55, "succeeded": 55, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 55 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · open-jev--mailroom-control-v1-choice-test-v1 · target_mass_at_prediction	100.00%	Setup & source S1MB datasets/open_jev__mailroom-control-v1 / test · target_mass_at_prediction. Population 15 cases / 15 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 15, "expected": 15, "succeeded": 15, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 15 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · open-jev--mailroom-control-v1-noul-test-v1 · binary_brier	0.0053 Brier	Setup & source S1MB datasets/open_jev__mailroom-control-v1 / test · binary_brier. Population 85 cases / 85 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 85, "expected": 85, "succeeded": 85, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 85 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · open-jev--painting-geometry-v1-choice-test-v1 · target_mass_at_prediction	88.00%	Setup & source S1MB datasets/open_jev__painting-geometry-v1 / test · target_mass_at_prediction. Population 25 cases / 25 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 25, "expected": 25, "succeeded": 25, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 25 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · open-jev--painting-geometry-v1-noul-test-v1 · binary_brier	0.2861 Brier	Setup & source S1MB datasets/open_jev__painting-geometry-v1 / test · binary_brier. Population 50 cases / 50 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 50, "expected": 50, "succeeded": 50, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 50 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · open-jev--painting-geometry-v1-score-test-v1 · normalized_expected_score_mae	0.1260 MAE	Setup & source S1MB datasets/open_jev__painting-geometry-v1 / test · normalized_expected_score_mae. Population 25 cases / 25 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 25, "expected": 25, "succeeded": 25, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 25 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · open-jev--phone-extraction-control-v1-choice-test-v1 · target_mass_at_prediction	83.54%	Setup & source S1MB datasets/open_jev__phone-extraction-control-v1 / test · target_mass_at_prediction. Population 79 cases / 79 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 79, "expected": 79, "succeeded": 79, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 79 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · open-jev--phone-extraction-control-v1-noul-test-v1 · binary_brier	0.0003 Brier	Setup & source S1MB datasets/open_jev__phone-extraction-control-v1 / test · binary_brier. Population 21 cases / 21 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 21, "expected": 21, "succeeded": 21, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 21 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · open-jev--reasoning-control-v1-choice-test-v1 · target_mass_at_prediction	100.00%	Setup & source S1MB datasets/open_jev__reasoning-control-v1 / test · target_mass_at_prediction. Population 33 cases / 33 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 33, "expected": 33, "succeeded": 33, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 33 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · open-jev--reasoning-control-v1-noul-test-v1 · binary_brier	0.0177 Brier	Setup & source S1MB datasets/open_jev__reasoning-control-v1 / test · binary_brier. Population 33 cases / 33 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 33, "expected": 33, "succeeded": 33, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 33 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · open-jev--reasoning-control-v1-score-test-v1 · normalized_expected_score_mae	0.0521 MAE	Setup & source S1MB datasets/open_jev__reasoning-control-v1 / test · normalized_expected_score_mae. Population 34 cases / 34 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 34, "expected": 34, "succeeded": 34, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 34 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · open-jev--silent-failure-control-v1-noul-test-v1 · binary_brier	0.0290 Brier	Setup & source S1MB datasets/open_jev__silent-failure-control-v1 / test · binary_brier. Population 100 cases / 100 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 100 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · open-jev--snake-v1-choice-test-v1 · target_mass_at_prediction	63.33%	Setup & source S1MB datasets/open_jev__snake-v1 / test · target_mass_at_prediction. Population 25 cases / 25 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 25, "expected": 25, "succeeded": 25, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 25 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · open-jev--snake-v1-noul-test-v1 · binary_brier	0.1746 Brier	Setup & source S1MB datasets/open_jev__snake-v1 / test · binary_brier. Population 75 cases / 75 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 75, "expected": 75, "succeeded": 75, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 75 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · open-jev--sponsor-segment-control-v1-choice-test-v1 · target_mass_at_prediction	96.00%	Setup & source S1MB datasets/open_jev__sponsor-segment-control-v1 / test · target_mass_at_prediction. Population 100 cases / 100 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 100 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · open-jev--tic-tac-toe-v1-choice-test-v1 · target_mass_at_prediction	60.00%	Setup & source S1MB datasets/open_jev__tic_tac_toe-v1 / test · target_mass_at_prediction. Population 100 cases / 100 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 100 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · open-jev--vizdoom-basic-v1-noul-test-v1 · binary_brier	0.0311 Brier	Setup & source S1MB datasets/open_jev__vizdoom-basic-v1 / test · binary_brier. Population 33 cases / 33 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 33, "expected": 33, "succeeded": 33, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 33 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · open-jev--vizdoom-basic-v1-score-test-v1 · normalized_expected_score_mae	0.0518 MAE	Setup & source S1MB datasets/open_jev__vizdoom-basic-v1 / test · normalized_expected_score_mae. Population 33 cases / 33 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 33, "expected": 33, "succeeded": 33, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 33 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · open-jev--workflow-controls-v1--agent-trace-observability-noul-test-v1 · binary_brier	0.1777 Brier	Setup & source S1MB datasets/open_jev__workflow-controls-v1__agent_trace_observability / test · binary_brier. Population 100 cases / 100 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 100 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · open-jev--workflow-controls-v1--customer-service-noul-test-v1 · binary_brier	0.1899 Brier	Setup & source S1MB datasets/open_jev__workflow-controls-v1__customer_service / test · binary_brier. Population 100 cases / 100 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 100 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · open-jev--workflow-controls-v1--invoice-processing-noul-test-v1 · binary_brier	0.0776 Brier	Setup & source S1MB datasets/open_jev__workflow-controls-v1__invoice_processing / test · binary_brier. Population 100 cases / 100 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 100 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · open-jev--workflow-controls-v1--security-incidents-noul-test-v1 · binary_brier	0.0633 Brier	Setup & source S1MB datasets/open_jev__workflow-controls-v1__security_incidents / test · binary_brier. Population 100 cases / 100 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 100 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · openbookqa-choice-test-v1 · target_mass_at_prediction	99.00%	Setup & source S1MB datasets/openbookqa / test · target_mass_at_prediction. Population 100 cases / 100 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 100 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · patent-similarity-score-test-v1 · normalized_expected_score_mae	0.1461 MAE	Setup & source S1MB datasets/patent_similarity / test · normalized_expected_score_mae. Population 100 cases / 100 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 100 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · paws-noul-test-v1 · binary_brier	0.1619 Brier	Setup & source S1MB datasets/paws / test · binary_brier. Population 100 cases / 100 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 100 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · plane-noul-test-v1 · binary_brier	0.3292 Brier	Setup & source S1MB datasets/plane / test · binary_brier. Population 100 cases / 100 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 100 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · poem-sentiment-choice-test-v1 · target_mass_at_prediction	55.00%	Setup & source S1MB datasets/poem_sentiment / test · target_mass_at_prediction. Population 100 cases / 100 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 100 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · pope-noul · binary_brier	0.1018 Brier	Setup & source S1MB datasets/pope-noul / test · binary_brier. Population 300 cases / 300 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 300, "expected": 300, "succeeded": 300, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 300 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · qasper-noul-test-v1 · binary_brier	0.0816 Brier	Setup & source S1MB datasets/qasper / test · binary_brier. Population 100 cases / 100 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 100 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · quartz-choice-test-v1 · target_mass_at_prediction	94.00%	Setup & source S1MB datasets/quartz / test · target_mass_at_prediction. Population 100 cases / 100 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 100 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · robust-lr-choice-test-v1 · target_mass_at_prediction	84.00%	Setup & source S1MB datasets/robust_lr / test · target_mass_at_prediction. Population 100 cases / 100 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 100 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · ruletaker-noul-test-v1 · binary_brier	0.1471 Brier	Setup & source S1MB datasets/ruletaker / test · binary_brier. Population 100 cases / 100 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 100 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · rvlcdip-choice · target_mass_at_prediction	72.00%	Setup & source S1MB datasets/rvlcdip-choice / test · target_mass_at_prediction. Population 300 cases / 300 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 300, "expected": 300, "succeeded": 300, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 300 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · s1mb-generalization-contextual-choice-choice-test-v1 · target_mass_at_prediction	99.00%	Setup & source S1MB datasets/s1mb-generalization-contextual-choice / test · target_mass_at_prediction. Population 100 cases / 100 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 100 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · s1mb-generalization-contextual-noul-noul-test-v1 · binary_brier	0.0303 Brier	Setup & source S1MB datasets/s1mb-generalization-contextual-noul / test · binary_brier. Population 100 cases / 100 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 100 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · s1mb-generalization-contextual-score-score-test-v1 · normalized_expected_score_mae	0.0266 MAE	Setup & source S1MB datasets/s1mb-generalization-contextual-score / test · normalized_expected_score_mae. Population 100 cases / 100 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 100 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · s1mb-generalization-diverse-choice-choice-test-v1 · target_mass_at_prediction	99.00%	Setup & source S1MB datasets/s1mb-generalization-diverse-choice / test · target_mass_at_prediction. Population 100 cases / 100 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 100 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · s1mb-generalization-diverse-noul-noul-test-v1 · binary_brier	0.0083 Brier	Setup & source S1MB datasets/s1mb-generalization-diverse-noul / test · binary_brier. Population 100 cases / 100 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 100 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · s1mb-generalization-diverse-score-score-test-v1 · normalized_expected_score_mae	0.0220 MAE	Setup & source S1MB datasets/s1mb-generalization-diverse-score / test · normalized_expected_score_mae. Population 100 cases / 100 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 100 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · scicite-choice-test-v1 · target_mass_at_prediction	75.00%	Setup & source S1MB datasets/scicite / test · target_mass_at_prediction. Population 100 cases / 100 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 100 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · scienceqa-choice · target_mass_at_prediction	95.00%	Setup & source S1MB datasets/scienceqa-choice / test · target_mass_at_prediction. Population 300 cases / 300 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 300, "expected": 300, "succeeded": 300, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 300 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · scitail-noul-test-v1 · binary_brier	0.0840 Brier	Setup & source S1MB datasets/scitail / test · binary_brier. Population 100 cases / 100 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 100 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · scone-choice-test-v1 · target_mass_at_prediction	94.00%	Setup & source S1MB datasets/scone / test · target_mass_at_prediction. Population 100 cases / 100 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 100 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · sdoh-nli-noul-test-v1 · binary_brier	0.0351 Brier	Setup & source S1MB datasets/sdoh_nli / test · binary_brier. Population 100 cases / 100 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 100 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · sgd-choice-test-v1 · target_mass_at_prediction	59.00%	Setup & source S1MB datasets/sgd / test · target_mass_at_prediction. Population 100 cases / 100 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 100 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · snli-choice-test-v1 · target_mass_at_prediction	79.00%	Setup & source S1MB datasets/snli / test · target_mass_at_prediction. Population 100 cases / 100 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 100 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · sobaco-culture-choice · target_mass_at_prediction	87.00%	Setup & source S1MB datasets/sobaco-culture-choice / test · target_mass_at_prediction. Population 300 cases / 300 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 300, "expected": 300, "succeeded": 300, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 300 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · spartqa-choice-test-v1 · target_mass_at_prediction	48.00%	Setup & source S1MB datasets/spartqa / test · target_mass_at_prediction. Population 100 cases / 100 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 100 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · stepgame-choice-test-v1 · target_mass_at_prediction	48.00%	Setup & source S1MB datasets/stepgame / test · target_mass_at_prediction. Population 100 cases / 100 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 100 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · sugarcrepe-choice · target_mass_at_prediction	96.33%	Setup & source S1MB datasets/sugarcrepe-choice / test · target_mass_at_prediction. Population 300 cases / 300 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 300, "expected": 300, "succeeded": 300, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 300 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · synthetic-relevance--nanobeir--nanoarguana-noul-test-v1 · binary_brier	0.1145 Brier	Setup & source S1MB datasets/synthetic_relevance__nanobeir__NanoArguAna / test · binary_brier. Population 400 cases / 400 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 400, "expected": 400, "succeeded": 400, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 400 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · synthetic-relevance--nanobeir--nanoarguana-score-test-v1 · normalized_expected_score_mae	0.1562 MAE	Setup & source S1MB datasets/synthetic_relevance__nanobeir__NanoArguAna / test · normalized_expected_score_mae. Population 400 cases / 400 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 400, "expected": 400, "succeeded": 400, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 400 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · synthetic-relevance--nanobeir--nanodbpedia-noul-test-v1 · binary_brier	0.0769 Brier	Setup & source S1MB datasets/synthetic_relevance__nanobeir__NanoDBPedia / test · binary_brier. Population 400 cases / 400 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 400, "expected": 400, "succeeded": 400, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 400 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · synthetic-relevance--nanobeir--nanodbpedia-score-test-v1 · normalized_expected_score_mae	0.1490 MAE	Setup & source S1MB datasets/synthetic_relevance__nanobeir__NanoDBPedia / test · normalized_expected_score_mae. Population 400 cases / 400 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 400, "expected": 400, "succeeded": 400, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 400 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · synthetic-relevance--nanobeir--nanofiqa2018-noul-test-v1 · binary_brier	0.0719 Brier	Setup & source S1MB datasets/synthetic_relevance__nanobeir__NanoFiQA2018 / test · binary_brier. Population 400 cases / 400 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 400, "expected": 400, "succeeded": 400, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 400 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · synthetic-relevance--nanobeir--nanofiqa2018-score-test-v1 · normalized_expected_score_mae	0.1539 MAE	Setup & source S1MB datasets/synthetic_relevance__nanobeir__NanoFiQA2018 / test · normalized_expected_score_mae. Population 400 cases / 400 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 400, "expected": 400, "succeeded": 400, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 400 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · synthetic-relevance--nanobeir--nanohotpotqa-noul-test-v1 · binary_brier	0.0546 Brier	Setup & source S1MB datasets/synthetic_relevance__nanobeir__NanoHotpotQA / test · binary_brier. Population 400 cases / 400 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 400, "expected": 400, "succeeded": 400, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 400 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · synthetic-relevance--nanobeir--nanohotpotqa-score-test-v1 · normalized_expected_score_mae	0.1242 MAE	Setup & source S1MB datasets/synthetic_relevance__nanobeir__NanoHotpotQA / test · normalized_expected_score_mae. Population 400 cases / 400 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 400, "expected": 400, "succeeded": 400, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 400 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · synthetic-relevance--nanobeir--nanomsmarco-noul-test-v1 · binary_brier	0.0617 Brier	Setup & source S1MB datasets/synthetic_relevance__nanobeir__NanoMSMARCO / test · binary_brier. Population 400 cases / 400 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 400, "expected": 400, "succeeded": 400, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 400 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · synthetic-relevance--nanobeir--nanomsmarco-score-test-v1 · normalized_expected_score_mae	0.1292 MAE	Setup & source S1MB datasets/synthetic_relevance__nanobeir__NanoMSMARCO / test · normalized_expected_score_mae. Population 400 cases / 400 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 400, "expected": 400, "succeeded": 400, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 400 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · synthetic-relevance--nanobeir--nanonfcorpus-noul-test-v1 · binary_brier	0.0851 Brier	Setup & source S1MB datasets/synthetic_relevance__nanobeir__NanoNFCorpus / test · binary_brier. Population 400 cases / 400 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 400, "expected": 400, "succeeded": 400, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 400 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · synthetic-relevance--nanobeir--nanonfcorpus-score-test-v1 · normalized_expected_score_mae	0.1430 MAE	Setup & source S1MB datasets/synthetic_relevance__nanobeir__NanoNFCorpus / test · normalized_expected_score_mae. Population 400 cases / 400 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 400, "expected": 400, "succeeded": 400, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 400 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · synthetic-relevance--nanobeir--nanonq-noul-test-v1 · binary_brier	0.0388 Brier	Setup & source S1MB datasets/synthetic_relevance__nanobeir__NanoNQ / test · binary_brier. Population 400 cases / 400 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 400, "expected": 400, "succeeded": 400, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 400 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · synthetic-relevance--nanobeir--nanonq-score-test-v1 · normalized_expected_score_mae	0.1143 MAE	Setup & source S1MB datasets/synthetic_relevance__nanobeir__NanoNQ / test · normalized_expected_score_mae. Population 400 cases / 400 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 400, "expected": 400, "succeeded": 400, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 400 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · synthetic-relevance--nanobeir--nanoscidocs-noul-test-v1 · binary_brier	0.0709 Brier	Setup & source S1MB datasets/synthetic_relevance__nanobeir__NanoSCIDOCS / test · binary_brier. Population 400 cases / 400 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 400, "expected": 400, "succeeded": 400, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 400 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · synthetic-relevance--nanobeir--nanoscidocs-score-test-v1 · normalized_expected_score_mae	0.1526 MAE	Setup & source S1MB datasets/synthetic_relevance__nanobeir__NanoSCIDOCS / test · normalized_expected_score_mae. Population 400 cases / 400 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 400, "expected": 400, "succeeded": 400, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 400 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · synthetic-relevance--nanobeir--nanotouche2020-noul-test-v1 · binary_brier	0.1255 Brier	Setup & source S1MB datasets/synthetic_relevance__nanobeir__NanoTouche2020 / test · binary_brier. Population 392 cases / 392 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 392, "expected": 392, "succeeded": 392, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 392 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · synthetic-relevance--nanobeir--nanotouche2020-score-test-v1 · normalized_expected_score_mae	0.1756 MAE	Setup & source S1MB datasets/synthetic_relevance__nanobeir__NanoTouche2020 / test · normalized_expected_score_mae. Population 392 cases / 392 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 392, "expected": 392, "succeeded": 392, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 392 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · synthetic-relevance--nanocoir-noul-test-v1 · binary_brier	0.0366 Brier	Setup & source S1MB datasets/synthetic_relevance__nanocoir / test · binary_brier. Population 800 cases / 800 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 800, "expected": 800, "succeeded": 800, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 800 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · synthetic-relevance--nanocoir-score-test-v1 · normalized_expected_score_mae	0.1188 MAE	Setup & source S1MB datasets/synthetic_relevance__nanocoir / test · normalized_expected_score_mae. Population 800 cases / 800 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 800, "expected": 800, "succeeded": 800, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 800 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · tallyqa-score · normalized_expected_score_mae	0.0317 MAE	Setup & source S1MB datasets/tallyqa-score / test · normalized_expected_score_mae. Population 300 cases / 300 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 300, "expected": 300, "succeeded": 300, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 300 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · temporal-nli-choice-test-v1 · target_mass_at_prediction	85.00%	Setup & source S1MB datasets/temporal_nli / test · target_mass_at_prediction. Population 100 cases / 100 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 100 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · tracie-noul-test-v1 · binary_brier	0.2578 Brier	Setup & source S1MB datasets/tracie / test · binary_brier. Population 100 cases / 100 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 100 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · ud-ewt-choice-test-v1 · target_mass_at_prediction	60.04%	Setup & source S1MB datasets/ud_ewt / test · target_mass_at_prediction. Population 100 cases / 1509 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 100, "expected": 1509, "succeeded": 1509, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 100 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · visa-noul · binary_brier	0.3891 Brier	Setup & source S1MB datasets/visa-noul / test · binary_brier. Population 300 cases / 300 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 300, "expected": 300, "succeeded": 300, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 300 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · vsr-noul · binary_brier	0.1808 Brier	Setup & source S1MB datasets/vsr-noul / test · binary_brier. Population 300 cases / 300 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 300, "expected": 300, "succeeded": 300, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 300 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · winobias-choice-test-v1 · target_mass_at_prediction	96.00%	Setup & source S1MB datasets/winobias / test · target_mass_at_prediction. Population 100 cases / 100 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 100 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · winowhy-noul-test-v1 · binary_brier	0.1444 Brier	Setup & source S1MB datasets/winowhy / test · binary_brier. Population 100 cases / 100 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 100 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source
S1MB · wiqa-choice-test-v1 · target_mass_at_prediction	50.00%	Setup & source S1MB datasets/wiqa / test · target_mass_at_prediction. Population 100 cases / 100 decisions. Exact lihe07/span-02, which Hugging Face redirects to respanai/span-02, the registered repository; adapter span; settings {}. Run lihe07__span-02. Published complete measured S1MB summary from the display snapshot of 2026-11-02T00:55:18.893Z. Execution/submission status is not assumed independent. Native binary Brier/normalized expected-score MAE is an error, not a percent. Counts: {"cases": 100, "expected": 100, "succeeded": 100, "failed": 0, "abstained": 0}. Pinned public display snapshot 2026-10-17T12:53:13.764Z. Benchmark definitions keep dataset/split and task semantics. Native binary Brier and normalized expected-score MAE are errors, not percents. Failed, partial or missing results never become zero. Checked / reported: 2026-11-02 · Samples: 100 Snapshot: 2026-10-17T12:53:13.764Z · Revision: 706ef6637478b78e908be90088f68d8c8f0d52e4; 1112da5ce7417def6fa805f9d1c3160c86e64e9d for Span-02 · Published aggregate observations; result reuse terms not separately specified Result source · Benchmark source

Evaluator-published · Benchmark Heaven

Span-02 · Evaluator-published · Benchmark Heaven
Evaluation	Result	Evidence
Benchmark Heaven JevBench v1.8 · Capability Score	69.339	Setup & source Capability Score on the rendered board. Respan Span-02 · Open weights · LLM decoder. Board key respan-span-02; method tag v1.7. Measured on the Second regular draw (v1.6.43), 2026-10-14; put on the v1.6.1 scale by the draw's anchor offset (see How it works). Source link https://huggingface.co/lihe07/span-02. Rendered board row. Benchmark Heaven's rendered board, read 2 November 2026: 283 ranked systems · 1,500 decisions each (600 for 19 equated API re-runs) · 131 newer ranked rows tagged with their draw · 14 older rows dated separately. Newer rows are measured on dated draws and put on the v1.6.1 scale by the draw's anchor offset, so a row's draw is part of its identity. Distinct from Leanmcp JevBench, the community Decision Index and the stored Benchmark Heaven v1.6.1 feed. Cost is the source's model on a GPU-time basis for self-hosted rows, not a provider tariff. Self-hosted latency is timed on the GPU over loopback and API latency end to end with the network; the source says they are not directly comparable, so they stay in separate datasets. No Examenos recomputation. Checked / reported: 2026-11-02 · Evaluated: 2026-10-14 · Samples: Not published Snapshot: 2026-10-31 · Revision: Board v1.7.48, method v1.8, updated 2026-10-31; page sha256 df9093384c9cdf5854e065f71dd3abd83a127793f22f4ca70268e9981e0a0b17 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
Benchmark Heaven JevBench v1.8 · Intelligence	60.9	Setup & source Intelligence on the rendered board. Respan Span-02 · Open weights · LLM decoder. Board key respan-span-02; method tag v1.7. Measured on the Second regular draw (v1.6.43), 2026-10-14; put on the v1.6.1 scale by the draw's anchor offset (see How it works). Source link https://huggingface.co/lihe07/span-02. Rendered board row. Benchmark Heaven's rendered board, read 2 November 2026: 283 ranked systems · 1,500 decisions each (600 for 19 equated API re-runs) · 131 newer ranked rows tagged with their draw · 14 older rows dated separately. Newer rows are measured on dated draws and put on the v1.6.1 scale by the draw's anchor offset, so a row's draw is part of its identity. Distinct from Leanmcp JevBench, the community Decision Index and the stored Benchmark Heaven v1.6.1 feed. Cost is the source's model on a GPU-time basis for self-hosted rows, not a provider tariff. Self-hosted latency is timed on the GPU over loopback and API latency end to end with the network; the source says they are not directly comparable, so they stay in separate datasets. No Examenos recomputation. Checked / reported: 2026-11-02 · Evaluated: 2026-10-14 · Samples: Not published Snapshot: 2026-10-31 · Revision: Board v1.7.48, method v1.8, updated 2026-10-31; page sha256 df9093384c9cdf5854e065f71dd3abd83a127793f22f4ca70268e9981e0a0b17 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
Benchmark Heaven JevBench v1.8 · Calibration	77.8	Setup & source Calibration on the rendered board. Respan Span-02 · Open weights · LLM decoder. Board key respan-span-02; method tag v1.7. Measured on the Second regular draw (v1.6.43), 2026-10-14; put on the v1.6.1 scale by the draw's anchor offset (see How it works). Source link https://huggingface.co/lihe07/span-02. Rendered board row. Benchmark Heaven's rendered board, read 2 November 2026: 283 ranked systems · 1,500 decisions each (600 for 19 equated API re-runs) · 131 newer ranked rows tagged with their draw · 14 older rows dated separately. Newer rows are measured on dated draws and put on the v1.6.1 scale by the draw's anchor offset, so a row's draw is part of its identity. Distinct from Leanmcp JevBench, the community Decision Index and the stored Benchmark Heaven v1.6.1 feed. Cost is the source's model on a GPU-time basis for self-hosted rows, not a provider tariff. Self-hosted latency is timed on the GPU over loopback and API latency end to end with the network; the source says they are not directly comparable, so they stay in separate datasets. No Examenos recomputation. Checked / reported: 2026-11-02 · Evaluated: 2026-10-14 · Samples: Not published Snapshot: 2026-10-31 · Revision: Board v1.7.48, method v1.8, updated 2026-10-31; page sha256 df9093384c9cdf5854e065f71dd3abd83a127793f22f4ca70268e9981e0a0b17 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
Benchmark Heaven JevBench v1.8 · Cost Score	53.5	Setup & source Cost Score on the rendered board. Respan Span-02 · Open weights · LLM decoder. Board key respan-span-02; method tag v1.7. Measured on the Second regular draw (v1.6.43), 2026-10-14; put on the v1.6.1 scale by the draw's anchor offset (see How it works). Source link https://huggingface.co/lihe07/span-02. Rendered board row. Benchmark Heaven's rendered board, read 2 November 2026: 283 ranked systems · 1,500 decisions each (600 for 19 equated API re-runs) · 131 newer ranked rows tagged with their draw · 14 older rows dated separately. Newer rows are measured on dated draws and put on the v1.6.1 scale by the draw's anchor offset, so a row's draw is part of its identity. Distinct from Leanmcp JevBench, the community Decision Index and the stored Benchmark Heaven v1.6.1 feed. Cost is the source's model on a GPU-time basis for self-hosted rows, not a provider tariff. Self-hosted latency is timed on the GPU over loopback and API latency end to end with the network; the source says they are not directly comparable, so they stay in separate datasets. No Examenos recomputation. Checked / reported: 2026-11-02 · Evaluated: 2026-10-14 · Samples: Not published Snapshot: 2026-10-31 · Revision: Board v1.7.48, method v1.8, updated 2026-10-31; page sha256 df9093384c9cdf5854e065f71dd3abd83a127793f22f4ca70268e9981e0a0b17 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
Benchmark Heaven JevBench v1.8 · Modelled cost per 1,000 decisions	0.03540 USD/1k decisions	Setup & source Modelled cost per 1,000 decisions on the rendered board. Respan Span-02 · Open weights · LLM decoder. Board key respan-span-02; method tag v1.7. Measured on the Second regular draw (v1.6.43), 2026-10-14; put on the v1.6.1 scale by the draw's anchor offset (see How it works). Source link https://huggingface.co/lihe07/span-02. Rendered board row. Benchmark Heaven's rendered board, read 2 November 2026: 283 ranked systems · 1,500 decisions each (600 for 19 equated API re-runs) · 131 newer ranked rows tagged with their draw · 14 older rows dated separately. Newer rows are measured on dated draws and put on the v1.6.1 scale by the draw's anchor offset, so a row's draw is part of its identity. Distinct from Leanmcp JevBench, the community Decision Index and the stored Benchmark Heaven v1.6.1 feed. Cost is the source's model on a GPU-time basis for self-hosted rows, not a provider tariff. Self-hosted latency is timed on the GPU over loopback and API latency end to end with the network; the source says they are not directly comparable, so they stay in separate datasets. No Examenos recomputation. Checked / reported: 2026-11-02 · Evaluated: 2026-10-14 · Samples: Not published Snapshot: 2026-10-31 · Revision: Board v1.7.48, method v1.8, updated 2026-10-31; page sha256 df9093384c9cdf5854e065f71dd3abd83a127793f22f4ca70268e9981e0a0b17 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
Benchmark Heaven JevBench v1.8 · Self-hosted p50 on the GPU, loopback	0.031 s	Setup & source Self-hosted p50 on the GPU, loopback on the rendered board. Respan Span-02 · Open weights · LLM decoder. Board key respan-span-02; method tag v1.7. Measured on the Second regular draw (v1.6.43), 2026-10-14; put on the v1.6.1 scale by the draw's anchor offset (see How it works). Source link https://huggingface.co/lihe07/span-02. Rendered board row. Benchmark Heaven's rendered board, read 2 November 2026: 283 ranked systems · 1,500 decisions each (600 for 19 equated API re-runs) · 131 newer ranked rows tagged with their draw · 14 older rows dated separately. Newer rows are measured on dated draws and put on the v1.6.1 scale by the draw's anchor offset, so a row's draw is part of its identity. Distinct from Leanmcp JevBench, the community Decision Index and the stored Benchmark Heaven v1.6.1 feed. Cost is the source's model on a GPU-time basis for self-hosted rows, not a provider tariff. Self-hosted latency is timed on the GPU over loopback and API latency end to end with the network; the source says they are not directly comparable, so they stay in separate datasets. No Examenos recomputation. Checked / reported: 2026-11-02 · Evaluated: 2026-10-14 · Samples: Not published Snapshot: 2026-10-31 · Revision: Board v1.7.48, method v1.8, updated 2026-10-31; page sha256 df9093384c9cdf5854e065f71dd3abd83a127793f22f4ca70268e9981e0a0b17 · Published summary observations; upstream task/result terms retained Result source · Benchmark source
Benchmark Heaven JevBench v1.8 · Self-hosted estimate including network (2x + 150 ms)	0.21 s	Setup & source Self-hosted estimate including network (2x + 150 ms) on the rendered board. Respan Span-02 · Open weights · LLM decoder. Board key respan-span-02; method tag v1.7. Measured on the Second regular draw (v1.6.43), 2026-10-14; put on the v1.6.1 scale by the draw's anchor offset (see How it works). Source link https://huggingface.co/lihe07/span-02. Rendered board row. Benchmark Heaven's rendered board, read 2 November 2026: 283 ranked systems · 1,500 decisions each (600 for 19 equated API re-runs) · 131 newer ranked rows tagged with their draw · 14 older rows dated separately. Newer rows are measured on dated draws and put on the v1.6.1 scale by the draw's anchor offset, so a row's draw is part of its identity. Distinct from Leanmcp JevBench, the community Decision Index and the stored Benchmark Heaven v1.6.1 feed. Cost is the source's model on a GPU-time basis for self-hosted rows, not a provider tariff. Self-hosted latency is timed on the GPU over loopback and API latency end to end with the network; the source says they are not directly comparable, so they stay in separate datasets. No Examenos recomputation. Checked / reported: 2026-11-02 · Evaluated: 2026-10-14 · Samples: Not published Snapshot: 2026-10-31 · Revision: Board v1.7.48, method v1.8, updated 2026-10-31; page sha256 df9093384c9cdf5854e065f71dd3abd83a127793f22f4ca70268e9981e0a0b17 · Published summary observations; upstream task/result terms retained Result source · Benchmark source

Span-02 · Evaluator-published · Community Decision Index
Evaluation	Result	Evidence
HF text 0.4.1 · Full score	59.07	Setup & source HF text 0.4.1 · Full score Exact source engine span-02; {"kind": "full fine-tune", "technique": "autoregressive", "technique_name": "autoregressive", "served_params": 25805936206, "params_estimated": false, "params_from_name": false, "trained_bytes": null, "model_url": "https://huggingface.co/lihe07/span-02", "code_url": null, "base_model": "google/gemma-4-26B-A4B-it", "base_url": "https://huggingface.co/google/gemma-4-26B-A4B-it", "weights_repo": null, "served_checkpoint": null}. New in the 2026-10-14 PR batch (PR #171). Submitted public 0.3 run adopted (kit rescore 63.62, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving (NVFP4 experts, FP8 dense); decision agreement with the submitted rows 100.00% on 831 rows (bar 96%). Served quantized (NVFP4 experts, FP8 dense) by the author's own engine, so the quantized agreement bar applies. Gated short thinking on about 19% of questions with a deterministic think budget; serving settings were compared on Decision Index public samples (author-disclosed). Decision Index 0.4.1 (release-v4), source generated 2026-10-29T14:04:16Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Source notes: New in the 2026-10-14 PR batch (PR #171). Submitted public 0.3 run adopted (kit rescore 63.62, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving (NVFP4 experts, FP8 dense); decision agreement with the submitted rows 100.00% on 831 rows (bar 96%). Served quantized (NVFP4 experts, FP8 dense) by the author's own engine, so the quantized agreement bar applies. Gated short thinking on about 19% of questions with a deterministic think budget; serving settings were compared on Decision Index public samples (author-disclosed). Author public-run adoption/execution status and private lab setup are disclosed in configuration; evaluator-published does not mean every public run was independently executed. The board does not name a served checkpoint in this repository history. Community Decision Index 0.4.1: expanded/refreshed harder private new-domain tasks and equating; Full remains 60% private same-skill +40% private new domains, zero public contribution. This population revision is not a model upgrade/regression. Public/native/calibration data remain on their unchanged 0.4 definitions. Source publication/generation is not a model run date. Public submitted runs plus evaluator private measurements are Evaluator-published, not wholly independently executed. Checked / reported: 2026-11-02 · Samples: Not published Snapshot: 2026-10-22T05:40:06Z · Revision: d9962113a1acc95c2d7522c2babfb989afbeb034; c130faf65f82e81d3bdefb01e758c712dc5c5199 for the 2 November 2026 additions · Published observations; result reuse terms not specified Result source · Benchmark source
HF text 0.4 · Public index	63.62	Setup & source HF text 0.4 · Public index Exact source engine span-02; {"kind": "full fine-tune", "technique": "autoregressive", "technique_name": "autoregressive", "served_params": 25805936206, "params_estimated": false, "params_from_name": false, "trained_bytes": null, "model_url": "https://huggingface.co/lihe07/span-02", "code_url": null, "base_model": "google/gemma-4-26B-A4B-it", "base_url": "https://huggingface.co/google/gemma-4-26B-A4B-it", "weights_repo": null, "served_checkpoint": null}. New in the 2026-10-14 PR batch (PR #171). Submitted public 0.3 run adopted (kit rescore 63.62, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving (NVFP4 experts, FP8 dense); decision agreement with the submitted rows 100.00% on 831 rows (bar 96%). Served quantized (NVFP4 experts, FP8 dense) by the author's own engine, so the quantized agreement bar applies. Gated short thinking on about 19% of questions with a deterministic think budget; serving settings were compared on Decision Index public samples (author-disclosed). Decision Index 0.4.1 (release-v4), source generated 2026-10-29T14:04:16Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Source notes: New in the 2026-10-14 PR batch (PR #171). Submitted public 0.3 run adopted (kit rescore 63.62, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving (NVFP4 experts, FP8 dense); decision agreement with the submitted rows 100.00% on 831 rows (bar 96%). Served quantized (NVFP4 experts, FP8 dense) by the author's own engine, so the quantized agreement bar applies. Gated short thinking on about 19% of questions with a deterministic think budget; serving settings were compared on Decision Index public samples (author-disclosed). Author public-run adoption/execution status and private lab setup are disclosed in configuration; evaluator-published does not mean every public run was independently executed. The board does not name a served checkpoint in this repository history. Decision Index 0.4 (release-v4), source generated 2026-10-20T14:13:56Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Checked / reported: 2026-11-02 · Samples: Not published Snapshot: 2026-10-20T14:13:56Z · Revision: 133313cf8963f2b6bed960abb0152fd14cbb6731; c130faf65f82e81d3bdefb01e758c712dc5c5199 for the 2 November 2026 additions · Published observations; result reuse terms not specified Result source · Benchmark source
HF text 0.4 · Private same-skill component	59.32	Setup & source HF text 0.4 · Private same-skill component Exact source engine span-02; {"kind": "full fine-tune", "technique": "autoregressive", "technique_name": "autoregressive", "served_params": 25805936206, "params_estimated": false, "params_from_name": false, "trained_bytes": null, "model_url": "https://huggingface.co/lihe07/span-02", "code_url": null, "base_model": "google/gemma-4-26B-A4B-it", "base_url": "https://huggingface.co/google/gemma-4-26B-A4B-it", "weights_repo": null, "served_checkpoint": null}. New in the 2026-10-14 PR batch (PR #171). Submitted public 0.3 run adopted (kit rescore 63.62, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving (NVFP4 experts, FP8 dense); decision agreement with the submitted rows 100.00% on 831 rows (bar 96%). Served quantized (NVFP4 experts, FP8 dense) by the author's own engine, so the quantized agreement bar applies. Gated short thinking on about 19% of questions with a deterministic think budget; serving settings were compared on Decision Index public samples (author-disclosed). Decision Index 0.4.1 (release-v4), source generated 2026-10-29T14:04:16Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Source notes: New in the 2026-10-14 PR batch (PR #171). Submitted public 0.3 run adopted (kit rescore 63.62, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving (NVFP4 experts, FP8 dense); decision agreement with the submitted rows 100.00% on 831 rows (bar 96%). Served quantized (NVFP4 experts, FP8 dense) by the author's own engine, so the quantized agreement bar applies. Gated short thinking on about 19% of questions with a deterministic think budget; serving settings were compared on Decision Index public samples (author-disclosed). Author public-run adoption/execution status and private lab setup are disclosed in configuration; evaluator-published does not mean every public run was independently executed. The board does not name a served checkpoint in this repository history. Decision Index 0.4 (release-v4), source generated 2026-10-20T14:13:56Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Checked / reported: 2026-11-02 · Samples: Not published Snapshot: 2026-10-20T14:13:56Z · Revision: 133313cf8963f2b6bed960abb0152fd14cbb6731; c130faf65f82e81d3bdefb01e758c712dc5c5199 for the 2 November 2026 additions · Published observations; result reuse terms not specified Result source · Benchmark source
HF text 0.4.1 · Private new-domain component	45.84	Setup & source HF text 0.4.1 · Private new-domain component Exact source engine span-02; {"kind": "full fine-tune", "technique": "autoregressive", "technique_name": "autoregressive", "served_params": 25805936206, "params_estimated": false, "params_from_name": false, "trained_bytes": null, "model_url": "https://huggingface.co/lihe07/span-02", "code_url": null, "base_model": "google/gemma-4-26B-A4B-it", "base_url": "https://huggingface.co/google/gemma-4-26B-A4B-it", "weights_repo": null, "served_checkpoint": null}. New in the 2026-10-14 PR batch (PR #171). Submitted public 0.3 run adopted (kit rescore 63.62, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving (NVFP4 experts, FP8 dense); decision agreement with the submitted rows 100.00% on 831 rows (bar 96%). Served quantized (NVFP4 experts, FP8 dense) by the author's own engine, so the quantized agreement bar applies. Gated short thinking on about 19% of questions with a deterministic think budget; serving settings were compared on Decision Index public samples (author-disclosed). Decision Index 0.4.1 (release-v4), source generated 2026-10-29T14:04:16Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Source notes: New in the 2026-10-14 PR batch (PR #171). Submitted public 0.3 run adopted (kit rescore 63.62, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving (NVFP4 experts, FP8 dense); decision agreement with the submitted rows 100.00% on 831 rows (bar 96%). Served quantized (NVFP4 experts, FP8 dense) by the author's own engine, so the quantized agreement bar applies. Gated short thinking on about 19% of questions with a deterministic think budget; serving settings were compared on Decision Index public samples (author-disclosed). Author public-run adoption/execution status and private lab setup are disclosed in configuration; evaluator-published does not mean every public run was independently executed. The board does not name a served checkpoint in this repository history. Community Decision Index 0.4.1: expanded/refreshed harder private new-domain tasks and equating; Full remains 60% private same-skill +40% private new domains, zero public contribution. This population revision is not a model upgrade/regression. Public/native/calibration data remain on their unchanged 0.4 definitions. Source publication/generation is not a model run date. Public submitted runs plus evaluator private measurements are Evaluator-published, not wholly independently executed. Checked / reported: 2026-11-02 · Samples: Not published Snapshot: 2026-10-22T05:40:06Z · Revision: d9962113a1acc95c2d7522c2babfb989afbeb034; c130faf65f82e81d3bdefb01e758c712dc5c5199 for the 2 November 2026 additions · Published observations; result reuse terms not specified Result source · Benchmark source
HF text 0.4 · Raw public index	72.77	Setup & source HF text 0.4 · Raw public index Exact source engine span-02; {"kind": "full fine-tune", "technique": "autoregressive", "technique_name": "autoregressive", "served_params": 25805936206, "params_estimated": false, "params_from_name": false, "trained_bytes": null, "model_url": "https://huggingface.co/lihe07/span-02", "code_url": null, "base_model": "google/gemma-4-26B-A4B-it", "base_url": "https://huggingface.co/google/gemma-4-26B-A4B-it", "weights_repo": null, "served_checkpoint": null}. New in the 2026-10-14 PR batch (PR #171). Submitted public 0.3 run adopted (kit rescore 63.62, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving (NVFP4 experts, FP8 dense); decision agreement with the submitted rows 100.00% on 831 rows (bar 96%). Served quantized (NVFP4 experts, FP8 dense) by the author's own engine, so the quantized agreement bar applies. Gated short thinking on about 19% of questions with a deterministic think budget; serving settings were compared on Decision Index public samples (author-disclosed). Decision Index 0.4.1 (release-v4), source generated 2026-10-29T14:04:16Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Source notes: New in the 2026-10-14 PR batch (PR #171). Submitted public 0.3 run adopted (kit rescore 63.62, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving (NVFP4 experts, FP8 dense); decision agreement with the submitted rows 100.00% on 831 rows (bar 96%). Served quantized (NVFP4 experts, FP8 dense) by the author's own engine, so the quantized agreement bar applies. Gated short thinking on about 19% of questions with a deterministic think budget; serving settings were compared on Decision Index public samples (author-disclosed). Author public-run adoption/execution status and private lab setup are disclosed in configuration; evaluator-published does not mean every public run was independently executed. The board does not name a served checkpoint in this repository history. Decision Index 0.4 (release-v4), source generated 2026-10-20T14:13:56Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Checked / reported: 2026-11-02 · Samples: Not published Snapshot: 2026-10-20T14:13:56Z · Revision: 133313cf8963f2b6bed960abb0152fd14cbb6731; c130faf65f82e81d3bdefb01e758c712dc5c5199 for the 2 November 2026 additions · Published observations; result reuse terms not specified Result source · Benchmark source
HF text 0.4 · Public breadth index	62.76	Setup & source HF text 0.4 · Public breadth index Exact source engine span-02; {"kind": "full fine-tune", "technique": "autoregressive", "technique_name": "autoregressive", "served_params": 25805936206, "params_estimated": false, "params_from_name": false, "trained_bytes": null, "model_url": "https://huggingface.co/lihe07/span-02", "code_url": null, "base_model": "google/gemma-4-26B-A4B-it", "base_url": "https://huggingface.co/google/gemma-4-26B-A4B-it", "weights_repo": null, "served_checkpoint": null}. New in the 2026-10-14 PR batch (PR #171). Submitted public 0.3 run adopted (kit rescore 63.62, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving (NVFP4 experts, FP8 dense); decision agreement with the submitted rows 100.00% on 831 rows (bar 96%). Served quantized (NVFP4 experts, FP8 dense) by the author's own engine, so the quantized agreement bar applies. Gated short thinking on about 19% of questions with a deterministic think budget; serving settings were compared on Decision Index public samples (author-disclosed). Decision Index 0.4.1 (release-v4), source generated 2026-10-29T14:04:16Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Source notes: New in the 2026-10-14 PR batch (PR #171). Submitted public 0.3 run adopted (kit rescore 63.62, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving (NVFP4 experts, FP8 dense); decision agreement with the submitted rows 100.00% on 831 rows (bar 96%). Served quantized (NVFP4 experts, FP8 dense) by the author's own engine, so the quantized agreement bar applies. Gated short thinking on about 19% of questions with a deterministic think budget; serving settings were compared on Decision Index public samples (author-disclosed). Author public-run adoption/execution status and private lab setup are disclosed in configuration; evaluator-published does not mean every public run was independently executed. The board does not name a served checkpoint in this repository history. Decision Index 0.4 (release-v4), source generated 2026-10-20T14:13:56Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Checked / reported: 2026-11-02 · Samples: Not published Snapshot: 2026-10-20T14:13:56Z · Revision: 133313cf8963f2b6bed960abb0152fd14cbb6731; c130faf65f82e81d3bdefb01e758c712dc5c5199 for the 2 November 2026 additions · Published observations; result reuse terms not specified Result source · Benchmark source
HF text 0.4 · Equated private same-skill	58.95	Setup & source HF text 0.4 · Equated private same-skill Exact source engine span-02; {"kind": "full fine-tune", "technique": "autoregressive", "technique_name": "autoregressive", "served_params": 25805936206, "params_estimated": false, "params_from_name": false, "trained_bytes": null, "model_url": "https://huggingface.co/lihe07/span-02", "code_url": null, "base_model": "google/gemma-4-26B-A4B-it", "base_url": "https://huggingface.co/google/gemma-4-26B-A4B-it", "weights_repo": null, "served_checkpoint": null}. New in the 2026-10-14 PR batch (PR #171). Submitted public 0.3 run adopted (kit rescore 63.62, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving (NVFP4 experts, FP8 dense); decision agreement with the submitted rows 100.00% on 831 rows (bar 96%). Served quantized (NVFP4 experts, FP8 dense) by the author's own engine, so the quantized agreement bar applies. Gated short thinking on about 19% of questions with a deterministic think budget; serving settings were compared on Decision Index public samples (author-disclosed). Decision Index 0.4.1 (release-v4), source generated 2026-10-29T14:04:16Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Source notes: New in the 2026-10-14 PR batch (PR #171). Submitted public 0.3 run adopted (kit rescore 63.62, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving (NVFP4 experts, FP8 dense); decision agreement with the submitted rows 100.00% on 831 rows (bar 96%). Served quantized (NVFP4 experts, FP8 dense) by the author's own engine, so the quantized agreement bar applies. Gated short thinking on about 19% of questions with a deterministic think budget; serving settings were compared on Decision Index public samples (author-disclosed). Author public-run adoption/execution status and private lab setup are disclosed in configuration; evaluator-published does not mean every public run was independently executed. The board does not name a served checkpoint in this repository history. Decision Index 0.4 (release-v4), source generated 2026-10-20T14:13:56Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Checked / reported: 2026-11-02 · Samples: Not published Snapshot: 2026-10-20T14:13:56Z · Revision: 133313cf8963f2b6bed960abb0152fd14cbb6731; c130faf65f82e81d3bdefb01e758c712dc5c5199 for the 2 November 2026 additions · Published observations; result reuse terms not specified Result source · Benchmark source
HF text 0.4.1 · Equated private new domains	59.26	Setup & source HF text 0.4.1 · Equated private new domains Exact source engine span-02; {"kind": "full fine-tune", "technique": "autoregressive", "technique_name": "autoregressive", "served_params": 25805936206, "params_estimated": false, "params_from_name": false, "trained_bytes": null, "model_url": "https://huggingface.co/lihe07/span-02", "code_url": null, "base_model": "google/gemma-4-26B-A4B-it", "base_url": "https://huggingface.co/google/gemma-4-26B-A4B-it", "weights_repo": null, "served_checkpoint": null}. New in the 2026-10-14 PR batch (PR #171). Submitted public 0.3 run adopted (kit rescore 63.62, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving (NVFP4 experts, FP8 dense); decision agreement with the submitted rows 100.00% on 831 rows (bar 96%). Served quantized (NVFP4 experts, FP8 dense) by the author's own engine, so the quantized agreement bar applies. Gated short thinking on about 19% of questions with a deterministic think budget; serving settings were compared on Decision Index public samples (author-disclosed). Decision Index 0.4.1 (release-v4), source generated 2026-10-29T14:04:16Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Source notes: New in the 2026-10-14 PR batch (PR #171). Submitted public 0.3 run adopted (kit rescore 63.62, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving (NVFP4 experts, FP8 dense); decision agreement with the submitted rows 100.00% on 831 rows (bar 96%). Served quantized (NVFP4 experts, FP8 dense) by the author's own engine, so the quantized agreement bar applies. Gated short thinking on about 19% of questions with a deterministic think budget; serving settings were compared on Decision Index public samples (author-disclosed). Author public-run adoption/execution status and private lab setup are disclosed in configuration; evaluator-published does not mean every public run was independently executed. The board does not name a served checkpoint in this repository history. Community Decision Index 0.4.1: expanded/refreshed harder private new-domain tasks and equating; Full remains 60% private same-skill +40% private new domains, zero public contribution. This population revision is not a model upgrade/regression. Public/native/calibration data remain on their unchanged 0.4 definitions. Source publication/generation is not a model run date. Public submitted runs plus evaluator private measurements are Evaluator-published, not wholly independently executed. Checked / reported: 2026-11-02 · Samples: Not published Snapshot: 2026-10-22T05:40:06Z · Revision: d9962113a1acc95c2d7522c2babfb989afbeb034; c130faf65f82e81d3bdefb01e758c712dc5c5199 for the 2 November 2026 additions · Published observations; result reuse terms not specified Result source · Benchmark source
HF text 0.4 · Full contribution · public	0.00	Setup & source HF text 0.4 · Full contribution · public Exact source engine span-02; {"kind": "full fine-tune", "technique": "autoregressive", "technique_name": "autoregressive", "served_params": 25805936206, "params_estimated": false, "params_from_name": false, "trained_bytes": null, "model_url": "https://huggingface.co/lihe07/span-02", "code_url": null, "base_model": "google/gemma-4-26B-A4B-it", "base_url": "https://huggingface.co/google/gemma-4-26B-A4B-it", "weights_repo": null, "served_checkpoint": null}. New in the 2026-10-14 PR batch (PR #171). Submitted public 0.3 run adopted (kit rescore 63.62, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving (NVFP4 experts, FP8 dense); decision agreement with the submitted rows 100.00% on 831 rows (bar 96%). Served quantized (NVFP4 experts, FP8 dense) by the author's own engine, so the quantized agreement bar applies. Gated short thinking on about 19% of questions with a deterministic think budget; serving settings were compared on Decision Index public samples (author-disclosed). Decision Index 0.4.1 (release-v4), source generated 2026-10-29T14:04:16Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Source notes: New in the 2026-10-14 PR batch (PR #171). Submitted public 0.3 run adopted (kit rescore 63.62, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving (NVFP4 experts, FP8 dense); decision agreement with the submitted rows 100.00% on 831 rows (bar 96%). Served quantized (NVFP4 experts, FP8 dense) by the author's own engine, so the quantized agreement bar applies. Gated short thinking on about 19% of questions with a deterministic think budget; serving settings were compared on Decision Index public samples (author-disclosed). Author public-run adoption/execution status and private lab setup are disclosed in configuration; evaluator-published does not mean every public run was independently executed. Published weighted contribution; not a distinct benchmark execution. The board does not name a served checkpoint in this repository history. Decision Index 0.4 (release-v4), source generated 2026-10-20T14:13:56Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Checked / reported: 2026-11-02 · Samples: Not published Snapshot: 2026-10-20T14:13:56Z · Revision: 133313cf8963f2b6bed960abb0152fd14cbb6731; c130faf65f82e81d3bdefb01e758c712dc5c5199 for the 2 November 2026 additions · Published observations; result reuse terms not specified Result source · Benchmark source
HF text 0.4 · Full contribution · same_skills	35.37	Setup & source HF text 0.4 · Full contribution · same_skills Exact source engine span-02; {"kind": "full fine-tune", "technique": "autoregressive", "technique_name": "autoregressive", "served_params": 25805936206, "params_estimated": false, "params_from_name": false, "trained_bytes": null, "model_url": "https://huggingface.co/lihe07/span-02", "code_url": null, "base_model": "google/gemma-4-26B-A4B-it", "base_url": "https://huggingface.co/google/gemma-4-26B-A4B-it", "weights_repo": null, "served_checkpoint": null}. New in the 2026-10-14 PR batch (PR #171). Submitted public 0.3 run adopted (kit rescore 63.62, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving (NVFP4 experts, FP8 dense); decision agreement with the submitted rows 100.00% on 831 rows (bar 96%). Served quantized (NVFP4 experts, FP8 dense) by the author's own engine, so the quantized agreement bar applies. Gated short thinking on about 19% of questions with a deterministic think budget; serving settings were compared on Decision Index public samples (author-disclosed). Decision Index 0.4.1 (release-v4), source generated 2026-10-29T14:04:16Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Source notes: New in the 2026-10-14 PR batch (PR #171). Submitted public 0.3 run adopted (kit rescore 63.62, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving (NVFP4 experts, FP8 dense); decision agreement with the submitted rows 100.00% on 831 rows (bar 96%). Served quantized (NVFP4 experts, FP8 dense) by the author's own engine, so the quantized agreement bar applies. Gated short thinking on about 19% of questions with a deterministic think budget; serving settings were compared on Decision Index public samples (author-disclosed). Author public-run adoption/execution status and private lab setup are disclosed in configuration; evaluator-published does not mean every public run was independently executed. Published weighted contribution; not a distinct benchmark execution. The board does not name a served checkpoint in this repository history. Decision Index 0.4 (release-v4), source generated 2026-10-20T14:13:56Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Checked / reported: 2026-11-02 · Samples: Not published Snapshot: 2026-10-20T14:13:56Z · Revision: 133313cf8963f2b6bed960abb0152fd14cbb6731; c130faf65f82e81d3bdefb01e758c712dc5c5199 for the 2 November 2026 additions · Published observations; result reuse terms not specified Result source · Benchmark source
HF text 0.4.1 · Full contribution · new_domains	23.70	Setup & source HF text 0.4.1 · Full contribution · new_domains Exact source engine span-02; {"kind": "full fine-tune", "technique": "autoregressive", "technique_name": "autoregressive", "served_params": 25805936206, "params_estimated": false, "params_from_name": false, "trained_bytes": null, "model_url": "https://huggingface.co/lihe07/span-02", "code_url": null, "base_model": "google/gemma-4-26B-A4B-it", "base_url": "https://huggingface.co/google/gemma-4-26B-A4B-it", "weights_repo": null, "served_checkpoint": null}. New in the 2026-10-14 PR batch (PR #171). Submitted public 0.3 run adopted (kit rescore 63.62, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving (NVFP4 experts, FP8 dense); decision agreement with the submitted rows 100.00% on 831 rows (bar 96%). Served quantized (NVFP4 experts, FP8 dense) by the author's own engine, so the quantized agreement bar applies. Gated short thinking on about 19% of questions with a deterministic think budget; serving settings were compared on Decision Index public samples (author-disclosed). Decision Index 0.4.1 (release-v4), source generated 2026-10-29T14:04:16Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Source notes: New in the 2026-10-14 PR batch (PR #171). Submitted public 0.3 run adopted (kit rescore 63.62, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving (NVFP4 experts, FP8 dense); decision agreement with the submitted rows 100.00% on 831 rows (bar 96%). Served quantized (NVFP4 experts, FP8 dense) by the author's own engine, so the quantized agreement bar applies. Gated short thinking on about 19% of questions with a deterministic think budget; serving settings were compared on Decision Index public samples (author-disclosed). Author public-run adoption/execution status and private lab setup are disclosed in configuration; evaluator-published does not mean every public run was independently executed. Published weighted contribution; not a distinct benchmark execution. The board does not name a served checkpoint in this repository history. Community Decision Index 0.4.1: expanded/refreshed harder private new-domain tasks and equating; Full remains 60% private same-skill +40% private new domains, zero public contribution. This population revision is not a model upgrade/regression. Public/native/calibration data remain on their unchanged 0.4 definitions. Source publication/generation is not a model run date. Public submitted runs plus evaluator private measurements are Evaluator-published, not wholly independently executed. Checked / reported: 2026-11-02 · Samples: Not published Snapshot: 2026-10-22T05:40:06Z · Revision: d9962113a1acc95c2d7522c2babfb989afbeb034; c130faf65f82e81d3bdefb01e758c712dc5c5199 for the 2 November 2026 additions · Published observations; result reuse terms not specified Result source · Benchmark source
HF text 0.4 · Knowledge · raw	62.00	Setup & source HF text 0.4 · Knowledge · raw Exact source engine span-02; {"kind": "full fine-tune", "technique": "autoregressive", "technique_name": "autoregressive", "served_params": 25805936206, "params_estimated": false, "params_from_name": false, "trained_bytes": null, "model_url": "https://huggingface.co/lihe07/span-02", "code_url": null, "base_model": "google/gemma-4-26B-A4B-it", "base_url": "https://huggingface.co/google/gemma-4-26B-A4B-it", "weights_repo": null, "served_checkpoint": null}. New in the 2026-10-14 PR batch (PR #171). Submitted public 0.3 run adopted (kit rescore 63.62, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving (NVFP4 experts, FP8 dense); decision agreement with the submitted rows 100.00% on 831 rows (bar 96%). Served quantized (NVFP4 experts, FP8 dense) by the author's own engine, so the quantized agreement bar applies. Gated short thinking on about 19% of questions with a deterministic think budget; serving settings were compared on Decision Index public samples (author-disclosed). Decision Index 0.4.1 (release-v4), source generated 2026-10-29T14:04:16Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Source notes: New in the 2026-10-14 PR batch (PR #171). Submitted public 0.3 run adopted (kit rescore 63.62, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving (NVFP4 experts, FP8 dense); decision agreement with the submitted rows 100.00% on 831 rows (bar 96%). Served quantized (NVFP4 experts, FP8 dense) by the author's own engine, so the quantized agreement bar applies. Gated short thinking on about 19% of questions with a deterministic think budget; serving settings were compared on Decision Index public samples (author-disclosed). Author public-run adoption/execution status and private lab setup are disclosed in configuration; evaluator-published does not mean every public run was independently executed. Count is contributing benchmarks, not requests. Source area context: {"id": "knowledge", "raw": 0.62, "skill": 0.5197, "combined": 0.4499, "coverage": 1, "pending": 0, "n": 11} The board does not name a served checkpoint in this repository history. Decision Index 0.4 (release-v4), source generated 2026-10-20T14:13:56Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Checked / reported: 2026-11-02 · Samples: 11 Snapshot: 2026-10-20T14:13:56Z · Revision: 133313cf8963f2b6bed960abb0152fd14cbb6731; c130faf65f82e81d3bdefb01e758c712dc5c5199 for the 2 November 2026 additions · Published observations; result reuse terms not specified Result source · Benchmark source
HF text 0.4 · Knowledge · skill	51.97	Setup & source HF text 0.4 · Knowledge · skill Exact source engine span-02; {"kind": "full fine-tune", "technique": "autoregressive", "technique_name": "autoregressive", "served_params": 25805936206, "params_estimated": false, "params_from_name": false, "trained_bytes": null, "model_url": "https://huggingface.co/lihe07/span-02", "code_url": null, "base_model": "google/gemma-4-26B-A4B-it", "base_url": "https://huggingface.co/google/gemma-4-26B-A4B-it", "weights_repo": null, "served_checkpoint": null}. New in the 2026-10-14 PR batch (PR #171). Submitted public 0.3 run adopted (kit rescore 63.62, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving (NVFP4 experts, FP8 dense); decision agreement with the submitted rows 100.00% on 831 rows (bar 96%). Served quantized (NVFP4 experts, FP8 dense) by the author's own engine, so the quantized agreement bar applies. Gated short thinking on about 19% of questions with a deterministic think budget; serving settings were compared on Decision Index public samples (author-disclosed). Decision Index 0.4.1 (release-v4), source generated 2026-10-29T14:04:16Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Source notes: New in the 2026-10-14 PR batch (PR #171). Submitted public 0.3 run adopted (kit rescore 63.62, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving (NVFP4 experts, FP8 dense); decision agreement with the submitted rows 100.00% on 831 rows (bar 96%). Served quantized (NVFP4 experts, FP8 dense) by the author's own engine, so the quantized agreement bar applies. Gated short thinking on about 19% of questions with a deterministic think budget; serving settings were compared on Decision Index public samples (author-disclosed). Author public-run adoption/execution status and private lab setup are disclosed in configuration; evaluator-published does not mean every public run was independently executed. Count is contributing benchmarks, not requests. Source area context: {"id": "knowledge", "raw": 0.62, "skill": 0.5197, "combined": 0.4499, "coverage": 1, "pending": 0, "n": 11} The board does not name a served checkpoint in this repository history. Decision Index 0.4 (release-v4), source generated 2026-10-20T14:13:56Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Checked / reported: 2026-11-02 · Samples: 11 Snapshot: 2026-10-20T14:13:56Z · Revision: 133313cf8963f2b6bed960abb0152fd14cbb6731; c130faf65f82e81d3bdefb01e758c712dc5c5199 for the 2 November 2026 additions · Published observations; result reuse terms not specified Result source · Benchmark source
HF text 0.4 · Knowledge · combined	44.99	Setup & source HF text 0.4 · Knowledge · combined Exact source engine span-02; {"kind": "full fine-tune", "technique": "autoregressive", "technique_name": "autoregressive", "served_params": 25805936206, "params_estimated": false, "params_from_name": false, "trained_bytes": null, "model_url": "https://huggingface.co/lihe07/span-02", "code_url": null, "base_model": "google/gemma-4-26B-A4B-it", "base_url": "https://huggingface.co/google/gemma-4-26B-A4B-it", "weights_repo": null, "served_checkpoint": null}. New in the 2026-10-14 PR batch (PR #171). Submitted public 0.3 run adopted (kit rescore 63.62, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving (NVFP4 experts, FP8 dense); decision agreement with the submitted rows 100.00% on 831 rows (bar 96%). Served quantized (NVFP4 experts, FP8 dense) by the author's own engine, so the quantized agreement bar applies. Gated short thinking on about 19% of questions with a deterministic think budget; serving settings were compared on Decision Index public samples (author-disclosed). Decision Index 0.4.1 (release-v4), source generated 2026-10-29T14:04:16Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Source notes: New in the 2026-10-14 PR batch (PR #171). Submitted public 0.3 run adopted (kit rescore 63.62, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving (NVFP4 experts, FP8 dense); decision agreement with the submitted rows 100.00% on 831 rows (bar 96%). Served quantized (NVFP4 experts, FP8 dense) by the author's own engine, so the quantized agreement bar applies. Gated short thinking on about 19% of questions with a deterministic think budget; serving settings were compared on Decision Index public samples (author-disclosed). Author public-run adoption/execution status and private lab setup are disclosed in configuration; evaluator-published does not mean every public run was independently executed. Count is contributing benchmarks, not requests. Source area context: {"id": "knowledge", "raw": 0.62, "skill": 0.5197, "combined": 0.4499, "coverage": 1, "pending": 0, "n": 11} The board does not name a served checkpoint in this repository history. Decision Index 0.4 (release-v4), source generated 2026-10-20T14:13:56Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Checked / reported: 2026-11-02 · Samples: 11 Snapshot: 2026-10-20T14:13:56Z · Revision: 133313cf8963f2b6bed960abb0152fd14cbb6731; c130faf65f82e81d3bdefb01e758c712dc5c5199 for the 2 November 2026 additions · Published observations; result reuse terms not specified Result source · Benchmark source
HF text 0.4 · Language · raw	75.11	Setup & source HF text 0.4 · Language · raw Exact source engine span-02; {"kind": "full fine-tune", "technique": "autoregressive", "technique_name": "autoregressive", "served_params": 25805936206, "params_estimated": false, "params_from_name": false, "trained_bytes": null, "model_url": "https://huggingface.co/lihe07/span-02", "code_url": null, "base_model": "google/gemma-4-26B-A4B-it", "base_url": "https://huggingface.co/google/gemma-4-26B-A4B-it", "weights_repo": null, "served_checkpoint": null}. New in the 2026-10-14 PR batch (PR #171). Submitted public 0.3 run adopted (kit rescore 63.62, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving (NVFP4 experts, FP8 dense); decision agreement with the submitted rows 100.00% on 831 rows (bar 96%). Served quantized (NVFP4 experts, FP8 dense) by the author's own engine, so the quantized agreement bar applies. Gated short thinking on about 19% of questions with a deterministic think budget; serving settings were compared on Decision Index public samples (author-disclosed). Decision Index 0.4.1 (release-v4), source generated 2026-10-29T14:04:16Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Source notes: New in the 2026-10-14 PR batch (PR #171). Submitted public 0.3 run adopted (kit rescore 63.62, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving (NVFP4 experts, FP8 dense); decision agreement with the submitted rows 100.00% on 831 rows (bar 96%). Served quantized (NVFP4 experts, FP8 dense) by the author's own engine, so the quantized agreement bar applies. Gated short thinking on about 19% of questions with a deterministic think budget; serving settings were compared on Decision Index public samples (author-disclosed). Author public-run adoption/execution status and private lab setup are disclosed in configuration; evaluator-published does not mean every public run was independently executed. Count is contributing benchmarks, not requests. Source area context: {"id": "language", "raw": 0.7511, "skill": 0.6525, "combined": 0.5738, "coverage": 1, "pending": 0, "n": 9} The board does not name a served checkpoint in this repository history. Decision Index 0.4 (release-v4), source generated 2026-10-20T14:13:56Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Checked / reported: 2026-11-02 · Samples: 9 Snapshot: 2026-10-20T14:13:56Z · Revision: 133313cf8963f2b6bed960abb0152fd14cbb6731; c130faf65f82e81d3bdefb01e758c712dc5c5199 for the 2 November 2026 additions · Published observations; result reuse terms not specified Result source · Benchmark source
HF text 0.4 · Language · skill	65.25	Setup & source HF text 0.4 · Language · skill Exact source engine span-02; {"kind": "full fine-tune", "technique": "autoregressive", "technique_name": "autoregressive", "served_params": 25805936206, "params_estimated": false, "params_from_name": false, "trained_bytes": null, "model_url": "https://huggingface.co/lihe07/span-02", "code_url": null, "base_model": "google/gemma-4-26B-A4B-it", "base_url": "https://huggingface.co/google/gemma-4-26B-A4B-it", "weights_repo": null, "served_checkpoint": null}. New in the 2026-10-14 PR batch (PR #171). Submitted public 0.3 run adopted (kit rescore 63.62, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving (NVFP4 experts, FP8 dense); decision agreement with the submitted rows 100.00% on 831 rows (bar 96%). Served quantized (NVFP4 experts, FP8 dense) by the author's own engine, so the quantized agreement bar applies. Gated short thinking on about 19% of questions with a deterministic think budget; serving settings were compared on Decision Index public samples (author-disclosed). Decision Index 0.4.1 (release-v4), source generated 2026-10-29T14:04:16Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Source notes: New in the 2026-10-14 PR batch (PR #171). Submitted public 0.3 run adopted (kit rescore 63.62, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving (NVFP4 experts, FP8 dense); decision agreement with the submitted rows 100.00% on 831 rows (bar 96%). Served quantized (NVFP4 experts, FP8 dense) by the author's own engine, so the quantized agreement bar applies. Gated short thinking on about 19% of questions with a deterministic think budget; serving settings were compared on Decision Index public samples (author-disclosed). Author public-run adoption/execution status and private lab setup are disclosed in configuration; evaluator-published does not mean every public run was independently executed. Count is contributing benchmarks, not requests. Source area context: {"id": "language", "raw": 0.7511, "skill": 0.6525, "combined": 0.5738, "coverage": 1, "pending": 0, "n": 9} The board does not name a served checkpoint in this repository history. Decision Index 0.4 (release-v4), source generated 2026-10-20T14:13:56Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Checked / reported: 2026-11-02 · Samples: 9 Snapshot: 2026-10-20T14:13:56Z · Revision: 133313cf8963f2b6bed960abb0152fd14cbb6731; c130faf65f82e81d3bdefb01e758c712dc5c5199 for the 2 November 2026 additions · Published observations; result reuse terms not specified Result source · Benchmark source
HF text 0.4 · Language · combined	57.38	Setup & source HF text 0.4 · Language · combined Exact source engine span-02; {"kind": "full fine-tune", "technique": "autoregressive", "technique_name": "autoregressive", "served_params": 25805936206, "params_estimated": false, "params_from_name": false, "trained_bytes": null, "model_url": "https://huggingface.co/lihe07/span-02", "code_url": null, "base_model": "google/gemma-4-26B-A4B-it", "base_url": "https://huggingface.co/google/gemma-4-26B-A4B-it", "weights_repo": null, "served_checkpoint": null}. New in the 2026-10-14 PR batch (PR #171). Submitted public 0.3 run adopted (kit rescore 63.62, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving (NVFP4 experts, FP8 dense); decision agreement with the submitted rows 100.00% on 831 rows (bar 96%). Served quantized (NVFP4 experts, FP8 dense) by the author's own engine, so the quantized agreement bar applies. Gated short thinking on about 19% of questions with a deterministic think budget; serving settings were compared on Decision Index public samples (author-disclosed). Decision Index 0.4.1 (release-v4), source generated 2026-10-29T14:04:16Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Source notes: New in the 2026-10-14 PR batch (PR #171). Submitted public 0.3 run adopted (kit rescore 63.62, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving (NVFP4 experts, FP8 dense); decision agreement with the submitted rows 100.00% on 831 rows (bar 96%). Served quantized (NVFP4 experts, FP8 dense) by the author's own engine, so the quantized agreement bar applies. Gated short thinking on about 19% of questions with a deterministic think budget; serving settings were compared on Decision Index public samples (author-disclosed). Author public-run adoption/execution status and private lab setup are disclosed in configuration; evaluator-published does not mean every public run was independently executed. Count is contributing benchmarks, not requests. Source area context: {"id": "language", "raw": 0.7511, "skill": 0.6525, "combined": 0.5738, "coverage": 1, "pending": 0, "n": 9} The board does not name a served checkpoint in this repository history. Decision Index 0.4 (release-v4), source generated 2026-10-20T14:13:56Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Checked / reported: 2026-11-02 · Samples: 9 Snapshot: 2026-10-20T14:13:56Z · Revision: 133313cf8963f2b6bed960abb0152fd14cbb6731; c130faf65f82e81d3bdefb01e758c712dc5c5199 for the 2 November 2026 additions · Published observations; result reuse terms not specified Result source · Benchmark source
HF text 0.4 · Retrieval · raw	74.23	Setup & source HF text 0.4 · Retrieval · raw Exact source engine span-02; {"kind": "full fine-tune", "technique": "autoregressive", "technique_name": "autoregressive", "served_params": 25805936206, "params_estimated": false, "params_from_name": false, "trained_bytes": null, "model_url": "https://huggingface.co/lihe07/span-02", "code_url": null, "base_model": "google/gemma-4-26B-A4B-it", "base_url": "https://huggingface.co/google/gemma-4-26B-A4B-it", "weights_repo": null, "served_checkpoint": null}. New in the 2026-10-14 PR batch (PR #171). Submitted public 0.3 run adopted (kit rescore 63.62, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving (NVFP4 experts, FP8 dense); decision agreement with the submitted rows 100.00% on 831 rows (bar 96%). Served quantized (NVFP4 experts, FP8 dense) by the author's own engine, so the quantized agreement bar applies. Gated short thinking on about 19% of questions with a deterministic think budget; serving settings were compared on Decision Index public samples (author-disclosed). Decision Index 0.4.1 (release-v4), source generated 2026-10-29T14:04:16Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Source notes: New in the 2026-10-14 PR batch (PR #171). Submitted public 0.3 run adopted (kit rescore 63.62, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving (NVFP4 experts, FP8 dense); decision agreement with the submitted rows 100.00% on 831 rows (bar 96%). Served quantized (NVFP4 experts, FP8 dense) by the author's own engine, so the quantized agreement bar applies. Gated short thinking on about 19% of questions with a deterministic think budget; serving settings were compared on Decision Index public samples (author-disclosed). Author public-run adoption/execution status and private lab setup are disclosed in configuration; evaluator-published does not mean every public run was independently executed. Count is contributing benchmarks, not requests. Source area context: {"id": "retrieval", "raw": 0.7423, "skill": 0.6554, "combined": 0.6317, "coverage": 1, "pending": 0, "n": 6} The board does not name a served checkpoint in this repository history. Decision Index 0.4 (release-v4), source generated 2026-10-20T14:13:56Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Checked / reported: 2026-11-02 · Samples: 6 Snapshot: 2026-10-20T14:13:56Z · Revision: 133313cf8963f2b6bed960abb0152fd14cbb6731; c130faf65f82e81d3bdefb01e758c712dc5c5199 for the 2 November 2026 additions · Published observations; result reuse terms not specified Result source · Benchmark source
HF text 0.4 · Retrieval · skill	65.54	Setup & source HF text 0.4 · Retrieval · skill Exact source engine span-02; {"kind": "full fine-tune", "technique": "autoregressive", "technique_name": "autoregressive", "served_params": 25805936206, "params_estimated": false, "params_from_name": false, "trained_bytes": null, "model_url": "https://huggingface.co/lihe07/span-02", "code_url": null, "base_model": "google/gemma-4-26B-A4B-it", "base_url": "https://huggingface.co/google/gemma-4-26B-A4B-it", "weights_repo": null, "served_checkpoint": null}. New in the 2026-10-14 PR batch (PR #171). Submitted public 0.3 run adopted (kit rescore 63.62, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving (NVFP4 experts, FP8 dense); decision agreement with the submitted rows 100.00% on 831 rows (bar 96%). Served quantized (NVFP4 experts, FP8 dense) by the author's own engine, so the quantized agreement bar applies. Gated short thinking on about 19% of questions with a deterministic think budget; serving settings were compared on Decision Index public samples (author-disclosed). Decision Index 0.4.1 (release-v4), source generated 2026-10-29T14:04:16Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Source notes: New in the 2026-10-14 PR batch (PR #171). Submitted public 0.3 run adopted (kit rescore 63.62, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving (NVFP4 experts, FP8 dense); decision agreement with the submitted rows 100.00% on 831 rows (bar 96%). Served quantized (NVFP4 experts, FP8 dense) by the author's own engine, so the quantized agreement bar applies. Gated short thinking on about 19% of questions with a deterministic think budget; serving settings were compared on Decision Index public samples (author-disclosed). Author public-run adoption/execution status and private lab setup are disclosed in configuration; evaluator-published does not mean every public run was independently executed. Count is contributing benchmarks, not requests. Source area context: {"id": "retrieval", "raw": 0.7423, "skill": 0.6554, "combined": 0.6317, "coverage": 1, "pending": 0, "n": 6} The board does not name a served checkpoint in this repository history. Decision Index 0.4 (release-v4), source generated 2026-10-20T14:13:56Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Checked / reported: 2026-11-02 · Samples: 6 Snapshot: 2026-10-20T14:13:56Z · Revision: 133313cf8963f2b6bed960abb0152fd14cbb6731; c130faf65f82e81d3bdefb01e758c712dc5c5199 for the 2 November 2026 additions · Published observations; result reuse terms not specified Result source · Benchmark source
HF text 0.4 · Retrieval · combined	63.17	Setup & source HF text 0.4 · Retrieval · combined Exact source engine span-02; {"kind": "full fine-tune", "technique": "autoregressive", "technique_name": "autoregressive", "served_params": 25805936206, "params_estimated": false, "params_from_name": false, "trained_bytes": null, "model_url": "https://huggingface.co/lihe07/span-02", "code_url": null, "base_model": "google/gemma-4-26B-A4B-it", "base_url": "https://huggingface.co/google/gemma-4-26B-A4B-it", "weights_repo": null, "served_checkpoint": null}. New in the 2026-10-14 PR batch (PR #171). Submitted public 0.3 run adopted (kit rescore 63.62, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving (NVFP4 experts, FP8 dense); decision agreement with the submitted rows 100.00% on 831 rows (bar 96%). Served quantized (NVFP4 experts, FP8 dense) by the author's own engine, so the quantized agreement bar applies. Gated short thinking on about 19% of questions with a deterministic think budget; serving settings were compared on Decision Index public samples (author-disclosed). Decision Index 0.4.1 (release-v4), source generated 2026-10-29T14:04:16Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Source notes: New in the 2026-10-14 PR batch (PR #171). Submitted public 0.3 run adopted (kit rescore 63.62, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving (NVFP4 experts, FP8 dense); decision agreement with the submitted rows 100.00% on 831 rows (bar 96%). Served quantized (NVFP4 experts, FP8 dense) by the author's own engine, so the quantized agreement bar applies. Gated short thinking on about 19% of questions with a deterministic think budget; serving settings were compared on Decision Index public samples (author-disclosed). Author public-run adoption/execution status and private lab setup are disclosed in configuration; evaluator-published does not mean every public run was independently executed. Count is contributing benchmarks, not requests. Source area context: {"id": "retrieval", "raw": 0.7423, "skill": 0.6554, "combined": 0.6317, "coverage": 1, "pending": 0, "n": 6} The board does not name a served checkpoint in this repository history. Decision Index 0.4 (release-v4), source generated 2026-10-20T14:13:56Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Checked / reported: 2026-11-02 · Samples: 6 Snapshot: 2026-10-20T14:13:56Z · Revision: 133313cf8963f2b6bed960abb0152fd14cbb6731; c130faf65f82e81d3bdefb01e758c712dc5c5199 for the 2 November 2026 additions · Published observations; result reuse terms not specified Result source · Benchmark source
HF text 0.4 · Tools · raw	86.10	Setup & source HF text 0.4 · Tools · raw Exact source engine span-02; {"kind": "full fine-tune", "technique": "autoregressive", "technique_name": "autoregressive", "served_params": 25805936206, "params_estimated": false, "params_from_name": false, "trained_bytes": null, "model_url": "https://huggingface.co/lihe07/span-02", "code_url": null, "base_model": "google/gemma-4-26B-A4B-it", "base_url": "https://huggingface.co/google/gemma-4-26B-A4B-it", "weights_repo": null, "served_checkpoint": null}. New in the 2026-10-14 PR batch (PR #171). Submitted public 0.3 run adopted (kit rescore 63.62, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving (NVFP4 experts, FP8 dense); decision agreement with the submitted rows 100.00% on 831 rows (bar 96%). Served quantized (NVFP4 experts, FP8 dense) by the author's own engine, so the quantized agreement bar applies. Gated short thinking on about 19% of questions with a deterministic think budget; serving settings were compared on Decision Index public samples (author-disclosed). Decision Index 0.4.1 (release-v4), source generated 2026-10-29T14:04:16Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Source notes: New in the 2026-10-14 PR batch (PR #171). Submitted public 0.3 run adopted (kit rescore 63.62, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving (NVFP4 experts, FP8 dense); decision agreement with the submitted rows 100.00% on 831 rows (bar 96%). Served quantized (NVFP4 experts, FP8 dense) by the author's own engine, so the quantized agreement bar applies. Gated short thinking on about 19% of questions with a deterministic think budget; serving settings were compared on Decision Index public samples (author-disclosed). Author public-run adoption/execution status and private lab setup are disclosed in configuration; evaluator-published does not mean every public run was independently executed. Count is contributing benchmarks, not requests. Source area context: {"id": "tools", "raw": 0.861, "skill": 0.8391, "combined": 0.7635, "coverage": 1, "pending": 0, "n": 5} The board does not name a served checkpoint in this repository history. Decision Index 0.4 (release-v4), source generated 2026-10-20T14:13:56Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Checked / reported: 2026-11-02 · Samples: 5 Snapshot: 2026-10-20T14:13:56Z · Revision: 133313cf8963f2b6bed960abb0152fd14cbb6731; c130faf65f82e81d3bdefb01e758c712dc5c5199 for the 2 November 2026 additions · Published observations; result reuse terms not specified Result source · Benchmark source
HF text 0.4 · Tools · skill	83.91	Setup & source HF text 0.4 · Tools · skill Exact source engine span-02; {"kind": "full fine-tune", "technique": "autoregressive", "technique_name": "autoregressive", "served_params": 25805936206, "params_estimated": false, "params_from_name": false, "trained_bytes": null, "model_url": "https://huggingface.co/lihe07/span-02", "code_url": null, "base_model": "google/gemma-4-26B-A4B-it", "base_url": "https://huggingface.co/google/gemma-4-26B-A4B-it", "weights_repo": null, "served_checkpoint": null}. New in the 2026-10-14 PR batch (PR #171). Submitted public 0.3 run adopted (kit rescore 63.62, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving (NVFP4 experts, FP8 dense); decision agreement with the submitted rows 100.00% on 831 rows (bar 96%). Served quantized (NVFP4 experts, FP8 dense) by the author's own engine, so the quantized agreement bar applies. Gated short thinking on about 19% of questions with a deterministic think budget; serving settings were compared on Decision Index public samples (author-disclosed). Decision Index 0.4.1 (release-v4), source generated 2026-10-29T14:04:16Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Source notes: New in the 2026-10-14 PR batch (PR #171). Submitted public 0.3 run adopted (kit rescore 63.62, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving (NVFP4 experts, FP8 dense); decision agreement with the submitted rows 100.00% on 831 rows (bar 96%). Served quantized (NVFP4 experts, FP8 dense) by the author's own engine, so the quantized agreement bar applies. Gated short thinking on about 19% of questions with a deterministic think budget; serving settings were compared on Decision Index public samples (author-disclosed). Author public-run adoption/execution status and private lab setup are disclosed in configuration; evaluator-published does not mean every public run was independently executed. Count is contributing benchmarks, not requests. Source area context: {"id": "tools", "raw": 0.861, "skill": 0.8391, "combined": 0.7635, "coverage": 1, "pending": 0, "n": 5} The board does not name a served checkpoint in this repository history. Decision Index 0.4 (release-v4), source generated 2026-10-20T14:13:56Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Checked / reported: 2026-11-02 · Samples: 5 Snapshot: 2026-10-20T14:13:56Z · Revision: 133313cf8963f2b6bed960abb0152fd14cbb6731; c130faf65f82e81d3bdefb01e758c712dc5c5199 for the 2 November 2026 additions · Published observations; result reuse terms not specified Result source · Benchmark source
HF text 0.4 · Tools · combined	76.35	Setup & source HF text 0.4 · Tools · combined Exact source engine span-02; {"kind": "full fine-tune", "technique": "autoregressive", "technique_name": "autoregressive", "served_params": 25805936206, "params_estimated": false, "params_from_name": false, "trained_bytes": null, "model_url": "https://huggingface.co/lihe07/span-02", "code_url": null, "base_model": "google/gemma-4-26B-A4B-it", "base_url": "https://huggingface.co/google/gemma-4-26B-A4B-it", "weights_repo": null, "served_checkpoint": null}. New in the 2026-10-14 PR batch (PR #171). Submitted public 0.3 run adopted (kit rescore 63.62, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving (NVFP4 experts, FP8 dense); decision agreement with the submitted rows 100.00% on 831 rows (bar 96%). Served quantized (NVFP4 experts, FP8 dense) by the author's own engine, so the quantized agreement bar applies. Gated short thinking on about 19% of questions with a deterministic think budget; serving settings were compared on Decision Index public samples (author-disclosed). Decision Index 0.4.1 (release-v4), source generated 2026-10-29T14:04:16Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Source notes: New in the 2026-10-14 PR batch (PR #171). Submitted public 0.3 run adopted (kit rescore 63.62, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving (NVFP4 experts, FP8 dense); decision agreement with the submitted rows 100.00% on 831 rows (bar 96%). Served quantized (NVFP4 experts, FP8 dense) by the author's own engine, so the quantized agreement bar applies. Gated short thinking on about 19% of questions with a deterministic think budget; serving settings were compared on Decision Index public samples (author-disclosed). Author public-run adoption/execution status and private lab setup are disclosed in configuration; evaluator-published does not mean every public run was independently executed. Count is contributing benchmarks, not requests. Source area context: {"id": "tools", "raw": 0.861, "skill": 0.8391, "combined": 0.7635, "coverage": 1, "pending": 0, "n": 5} The board does not name a served checkpoint in this repository history. Decision Index 0.4 (release-v4), source generated 2026-10-20T14:13:56Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Checked / reported: 2026-11-02 · Samples: 5 Snapshot: 2026-10-20T14:13:56Z · Revision: 133313cf8963f2b6bed960abb0152fd14cbb6731; c130faf65f82e81d3bdefb01e758c712dc5c5199 for the 2 November 2026 additions · Published observations; result reuse terms not specified Result source · Benchmark source
HF text 0.4 · Arts · raw	67.25	Setup & source HF text 0.4 · Arts · raw Exact source engine span-02; {"kind": "full fine-tune", "technique": "autoregressive", "technique_name": "autoregressive", "served_params": 25805936206, "params_estimated": false, "params_from_name": false, "trained_bytes": null, "model_url": "https://huggingface.co/lihe07/span-02", "code_url": null, "base_model": "google/gemma-4-26B-A4B-it", "base_url": "https://huggingface.co/google/gemma-4-26B-A4B-it", "weights_repo": null, "served_checkpoint": null}. New in the 2026-10-14 PR batch (PR #171). Submitted public 0.3 run adopted (kit rescore 63.62, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving (NVFP4 experts, FP8 dense); decision agreement with the submitted rows 100.00% on 831 rows (bar 96%). Served quantized (NVFP4 experts, FP8 dense) by the author's own engine, so the quantized agreement bar applies. Gated short thinking on about 19% of questions with a deterministic think budget; serving settings were compared on Decision Index public samples (author-disclosed). Decision Index 0.4.1 (release-v4), source generated 2026-10-29T14:04:16Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Source notes: New in the 2026-10-14 PR batch (PR #171). Submitted public 0.3 run adopted (kit rescore 63.62, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving (NVFP4 experts, FP8 dense); decision agreement with the submitted rows 100.00% on 831 rows (bar 96%). Served quantized (NVFP4 experts, FP8 dense) by the author's own engine, so the quantized agreement bar applies. Gated short thinking on about 19% of questions with a deterministic think budget; serving settings were compared on Decision Index public samples (author-disclosed). Author public-run adoption/execution status and private lab setup are disclosed in configuration; evaluator-published does not mean every public run was independently executed. Count is contributing benchmarks, not requests. Source area context: {"id": "arts", "raw": 0.6725, "skill": 0.4863, "combined": 0.5395, "coverage": 1, "pending": 0, "n": 6} The board does not name a served checkpoint in this repository history. Decision Index 0.4 (release-v4), source generated 2026-10-20T14:13:56Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Checked / reported: 2026-11-02 · Samples: 6 Snapshot: 2026-10-20T14:13:56Z · Revision: 133313cf8963f2b6bed960abb0152fd14cbb6731; c130faf65f82e81d3bdefb01e758c712dc5c5199 for the 2 November 2026 additions · Published observations; result reuse terms not specified Result source · Benchmark source
HF text 0.4 · Arts · skill	48.63	Setup & source HF text 0.4 · Arts · skill Exact source engine span-02; {"kind": "full fine-tune", "technique": "autoregressive", "technique_name": "autoregressive", "served_params": 25805936206, "params_estimated": false, "params_from_name": false, "trained_bytes": null, "model_url": "https://huggingface.co/lihe07/span-02", "code_url": null, "base_model": "google/gemma-4-26B-A4B-it", "base_url": "https://huggingface.co/google/gemma-4-26B-A4B-it", "weights_repo": null, "served_checkpoint": null}. New in the 2026-10-14 PR batch (PR #171). Submitted public 0.3 run adopted (kit rescore 63.62, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving (NVFP4 experts, FP8 dense); decision agreement with the submitted rows 100.00% on 831 rows (bar 96%). Served quantized (NVFP4 experts, FP8 dense) by the author's own engine, so the quantized agreement bar applies. Gated short thinking on about 19% of questions with a deterministic think budget; serving settings were compared on Decision Index public samples (author-disclosed). Decision Index 0.4.1 (release-v4), source generated 2026-10-29T14:04:16Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Source notes: New in the 2026-10-14 PR batch (PR #171). Submitted public 0.3 run adopted (kit rescore 63.62, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving (NVFP4 experts, FP8 dense); decision agreement with the submitted rows 100.00% on 831 rows (bar 96%). Served quantized (NVFP4 experts, FP8 dense) by the author's own engine, so the quantized agreement bar applies. Gated short thinking on about 19% of questions with a deterministic think budget; serving settings were compared on Decision Index public samples (author-disclosed). Author public-run adoption/execution status and private lab setup are disclosed in configuration; evaluator-published does not mean every public run was independently executed. Count is contributing benchmarks, not requests. Source area context: {"id": "arts", "raw": 0.6725, "skill": 0.4863, "combined": 0.5395, "coverage": 1, "pending": 0, "n": 6} The board does not name a served checkpoint in this repository history. Decision Index 0.4 (release-v4), source generated 2026-10-20T14:13:56Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Checked / reported: 2026-11-02 · Samples: 6 Snapshot: 2026-10-20T14:13:56Z · Revision: 133313cf8963f2b6bed960abb0152fd14cbb6731; c130faf65f82e81d3bdefb01e758c712dc5c5199 for the 2 November 2026 additions · Published observations; result reuse terms not specified Result source · Benchmark source
HF text 0.4 · Arts · combined	53.95	Setup & source HF text 0.4 · Arts · combined Exact source engine span-02; {"kind": "full fine-tune", "technique": "autoregressive", "technique_name": "autoregressive", "served_params": 25805936206, "params_estimated": false, "params_from_name": false, "trained_bytes": null, "model_url": "https://huggingface.co/lihe07/span-02", "code_url": null, "base_model": "google/gemma-4-26B-A4B-it", "base_url": "https://huggingface.co/google/gemma-4-26B-A4B-it", "weights_repo": null, "served_checkpoint": null}. New in the 2026-10-14 PR batch (PR #171). Submitted public 0.3 run adopted (kit rescore 63.62, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving (NVFP4 experts, FP8 dense); decision agreement with the submitted rows 100.00% on 831 rows (bar 96%). Served quantized (NVFP4 experts, FP8 dense) by the author's own engine, so the quantized agreement bar applies. Gated short thinking on about 19% of questions with a deterministic think budget; serving settings were compared on Decision Index public samples (author-disclosed). Decision Index 0.4.1 (release-v4), source generated 2026-10-29T14:04:16Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Source notes: New in the 2026-10-14 PR batch (PR #171). Submitted public 0.3 run adopted (kit rescore 63.62, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving (NVFP4 experts, FP8 dense); decision agreement with the submitted rows 100.00% on 831 rows (bar 96%). Served quantized (NVFP4 experts, FP8 dense) by the author's own engine, so the quantized agreement bar applies. Gated short thinking on about 19% of questions with a deterministic think budget; serving settings were compared on Decision Index public samples (author-disclosed). Author public-run adoption/execution status and private lab setup are disclosed in configuration; evaluator-published does not mean every public run was independently executed. Count is contributing benchmarks, not requests. Source area context: {"id": "arts", "raw": 0.6725, "skill": 0.4863, "combined": 0.5395, "coverage": 1, "pending": 0, "n": 6} The board does not name a served checkpoint in this repository history. Decision Index 0.4 (release-v4), source generated 2026-10-20T14:13:56Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Checked / reported: 2026-11-02 · Samples: 6 Snapshot: 2026-10-20T14:13:56Z · Revision: 133313cf8963f2b6bed960abb0152fd14cbb6731; c130faf65f82e81d3bdefb01e758c712dc5c5199 for the 2 November 2026 additions · Published observations; result reuse terms not specified Result source · Benchmark source
HF text calibration 0.4 · ECE	11.40 pp	Setup & source HF text calibration 0.4 · ECE Exact source engine span-02; {"kind": "full fine-tune", "technique": "autoregressive", "technique_name": "autoregressive", "served_params": 25805936206, "params_estimated": false, "params_from_name": false, "trained_bytes": null, "model_url": "https://huggingface.co/lihe07/span-02", "code_url": null, "base_model": "google/gemma-4-26B-A4B-it", "base_url": "https://huggingface.co/google/gemma-4-26B-A4B-it", "weights_repo": null, "served_checkpoint": null}. New in the 2026-10-14 PR batch (PR #171). Submitted public 0.3 run adopted (kit rescore 63.62, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving (NVFP4 experts, FP8 dense); decision agreement with the submitted rows 100.00% on 831 rows (bar 96%). Served quantized (NVFP4 experts, FP8 dense) by the author's own engine, so the quantized agreement bar applies. Gated short thinking on about 19% of questions with a deterministic think budget; serving settings were compared on Decision Index public samples (author-disclosed). Decision Index 0.4.1 (release-v4), source generated 2026-10-29T14:04:16Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Source notes: New in the 2026-10-14 PR batch (PR #171). Submitted public 0.3 run adopted (kit rescore 63.62, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving (NVFP4 experts, FP8 dense); decision agreement with the submitted rows 100.00% on 831 rows (bar 96%). Served quantized (NVFP4 experts, FP8 dense) by the author's own engine, so the quantized agreement bar applies. Gated short thinking on about 19% of questions with a deterministic think budget; serving settings were compared on Decision Index public samples (author-disclosed). Author public-run adoption/execution status and private lab setup are disclosed in configuration; evaluator-published does not mean every public run was independently executed. 32 benchmarks with a per-field right or wrong answer (ToolRet, BRIGHT and RouterBench excluded), each weighted equally; a 1-in-6 sample of requests; confidence is the probability placed on the chosen option. Rows an entrant trained on count as wrong here too. Contributing benchmarks 31. The board does not name a served checkpoint in this repository history. Decision Index 0.4 (release-v4), source generated 2026-10-20T14:13:56Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Checked / reported: 2026-11-02 · Samples: 34411 Snapshot: 2026-10-20T14:13:56Z · Revision: 133313cf8963f2b6bed960abb0152fd14cbb6731; c130faf65f82e81d3bdefb01e758c712dc5c5199 for the 2 November 2026 additions · Published observations; result reuse terms not specified Result source · Benchmark source
HF text calibration 0.4 · Brier	0.3553 Brier	Setup & source HF text calibration 0.4 · Brier Exact source engine span-02; {"kind": "full fine-tune", "technique": "autoregressive", "technique_name": "autoregressive", "served_params": 25805936206, "params_estimated": false, "params_from_name": false, "trained_bytes": null, "model_url": "https://huggingface.co/lihe07/span-02", "code_url": null, "base_model": "google/gemma-4-26B-A4B-it", "base_url": "https://huggingface.co/google/gemma-4-26B-A4B-it", "weights_repo": null, "served_checkpoint": null}. New in the 2026-10-14 PR batch (PR #171). Submitted public 0.3 run adopted (kit rescore 63.62, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving (NVFP4 experts, FP8 dense); decision agreement with the submitted rows 100.00% on 831 rows (bar 96%). Served quantized (NVFP4 experts, FP8 dense) by the author's own engine, so the quantized agreement bar applies. Gated short thinking on about 19% of questions with a deterministic think budget; serving settings were compared on Decision Index public samples (author-disclosed). Decision Index 0.4.1 (release-v4), source generated 2026-10-29T14:04:16Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Source notes: New in the 2026-10-14 PR batch (PR #171). Submitted public 0.3 run adopted (kit rescore 63.62, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving (NVFP4 experts, FP8 dense); decision agreement with the submitted rows 100.00% on 831 rows (bar 96%). Served quantized (NVFP4 experts, FP8 dense) by the author's own engine, so the quantized agreement bar applies. Gated short thinking on about 19% of questions with a deterministic think budget; serving settings were compared on Decision Index public samples (author-disclosed). Author public-run adoption/execution status and private lab setup are disclosed in configuration; evaluator-published does not mean every public run was independently executed. 32 benchmarks with a per-field right or wrong answer (ToolRet, BRIGHT and RouterBench excluded), each weighted equally; a 1-in-6 sample of requests; confidence is the probability placed on the chosen option. Rows an entrant trained on count as wrong here too. Contributing benchmarks 31. The board does not name a served checkpoint in this repository history. Decision Index 0.4 (release-v4), source generated 2026-10-20T14:13:56Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Checked / reported: 2026-11-02 · Samples: 34411 Snapshot: 2026-10-20T14:13:56Z · Revision: 133313cf8963f2b6bed960abb0152fd14cbb6731; c130faf65f82e81d3bdefb01e758c712dc5c5199 for the 2 November 2026 additions · Published observations; result reuse terms not specified Result source · Benchmark source
HF text calibration 0.4 · Sample accuracy	76.21%	Setup & source HF text calibration 0.4 · Sample accuracy Exact source engine span-02; {"kind": "full fine-tune", "technique": "autoregressive", "technique_name": "autoregressive", "served_params": 25805936206, "params_estimated": false, "params_from_name": false, "trained_bytes": null, "model_url": "https://huggingface.co/lihe07/span-02", "code_url": null, "base_model": "google/gemma-4-26B-A4B-it", "base_url": "https://huggingface.co/google/gemma-4-26B-A4B-it", "weights_repo": null, "served_checkpoint": null}. New in the 2026-10-14 PR batch (PR #171). Submitted public 0.3 run adopted (kit rescore 63.62, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving (NVFP4 experts, FP8 dense); decision agreement with the submitted rows 100.00% on 831 rows (bar 96%). Served quantized (NVFP4 experts, FP8 dense) by the author's own engine, so the quantized agreement bar applies. Gated short thinking on about 19% of questions with a deterministic think budget; serving settings were compared on Decision Index public samples (author-disclosed). Decision Index 0.4.1 (release-v4), source generated 2026-10-29T14:04:16Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Source notes: New in the 2026-10-14 PR batch (PR #171). Submitted public 0.3 run adopted (kit rescore 63.62, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving (NVFP4 experts, FP8 dense); decision agreement with the submitted rows 100.00% on 831 rows (bar 96%). Served quantized (NVFP4 experts, FP8 dense) by the author's own engine, so the quantized agreement bar applies. Gated short thinking on about 19% of questions with a deterministic think budget; serving settings were compared on Decision Index public samples (author-disclosed). Author public-run adoption/execution status and private lab setup are disclosed in configuration; evaluator-published does not mean every public run was independently executed. 32 benchmarks with a per-field right or wrong answer (ToolRet, BRIGHT and RouterBench excluded), each weighted equally; a 1-in-6 sample of requests; confidence is the probability placed on the chosen option. Rows an entrant trained on count as wrong here too. Contributing benchmarks 31. The board does not name a served checkpoint in this repository history. Decision Index 0.4 (release-v4), source generated 2026-10-20T14:13:56Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Checked / reported: 2026-11-02 · Samples: 34411 Snapshot: 2026-10-20T14:13:56Z · Revision: 133313cf8963f2b6bed960abb0152fd14cbb6731; c130faf65f82e81d3bdefb01e758c712dc5c5199 for the 2 November 2026 additions · Published observations; result reuse terms not specified Result source · Benchmark source
HF text calibration 0.4 · Mean chosen confidence	87.61%	Setup & source HF text calibration 0.4 · Mean chosen confidence Exact source engine span-02; {"kind": "full fine-tune", "technique": "autoregressive", "technique_name": "autoregressive", "served_params": 25805936206, "params_estimated": false, "params_from_name": false, "trained_bytes": null, "model_url": "https://huggingface.co/lihe07/span-02", "code_url": null, "base_model": "google/gemma-4-26B-A4B-it", "base_url": "https://huggingface.co/google/gemma-4-26B-A4B-it", "weights_repo": null, "served_checkpoint": null}. New in the 2026-10-14 PR batch (PR #171). Submitted public 0.3 run adopted (kit rescore 63.62, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving (NVFP4 experts, FP8 dense); decision agreement with the submitted rows 100.00% on 831 rows (bar 96%). Served quantized (NVFP4 experts, FP8 dense) by the author's own engine, so the quantized agreement bar applies. Gated short thinking on about 19% of questions with a deterministic think budget; serving settings were compared on Decision Index public samples (author-disclosed). Decision Index 0.4.1 (release-v4), source generated 2026-10-29T14:04:16Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Source notes: New in the 2026-10-14 PR batch (PR #171). Submitted public 0.3 run adopted (kit rescore 63.62, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving (NVFP4 experts, FP8 dense); decision agreement with the submitted rows 100.00% on 831 rows (bar 96%). Served quantized (NVFP4 experts, FP8 dense) by the author's own engine, so the quantized agreement bar applies. Gated short thinking on about 19% of questions with a deterministic think budget; serving settings were compared on Decision Index public samples (author-disclosed). Author public-run adoption/execution status and private lab setup are disclosed in configuration; evaluator-published does not mean every public run was independently executed. 32 benchmarks with a per-field right or wrong answer (ToolRet, BRIGHT and RouterBench excluded), each weighted equally; a 1-in-6 sample of requests; confidence is the probability placed on the chosen option. Rows an entrant trained on count as wrong here too. Contributing benchmarks 31. The board does not name a served checkpoint in this repository history. Decision Index 0.4 (release-v4), source generated 2026-10-20T14:13:56Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Checked / reported: 2026-11-02 · Samples: 34411 Snapshot: 2026-10-20T14:13:56Z · Revision: 133313cf8963f2b6bed960abb0152fd14cbb6731; c130faf65f82e81d3bdefb01e758c712dc5c5199 for the 2 November 2026 additions · Published observations; result reuse terms not specified Result source · Benchmark source
HF text calibration 0.4 · Wrong among answers at ≥95% confidence	2.77%	Setup & source HF text calibration 0.4 · Wrong among answers at ≥95% confidence Exact source engine span-02; {"kind": "full fine-tune", "technique": "autoregressive", "technique_name": "autoregressive", "served_params": 25805936206, "params_estimated": false, "params_from_name": false, "trained_bytes": null, "model_url": "https://huggingface.co/lihe07/span-02", "code_url": null, "base_model": "google/gemma-4-26B-A4B-it", "base_url": "https://huggingface.co/google/gemma-4-26B-A4B-it", "weights_repo": null, "served_checkpoint": null}. New in the 2026-10-14 PR batch (PR #171). Submitted public 0.3 run adopted (kit rescore 63.62, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving (NVFP4 experts, FP8 dense); decision agreement with the submitted rows 100.00% on 831 rows (bar 96%). Served quantized (NVFP4 experts, FP8 dense) by the author's own engine, so the quantized agreement bar applies. Gated short thinking on about 19% of questions with a deterministic think budget; serving settings were compared on Decision Index public samples (author-disclosed). Decision Index 0.4.1 (release-v4), source generated 2026-10-29T14:04:16Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Source notes: New in the 2026-10-14 PR batch (PR #171). Submitted public 0.3 run adopted (kit rescore 63.62, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving (NVFP4 experts, FP8 dense); decision agreement with the submitted rows 100.00% on 831 rows (bar 96%). Served quantized (NVFP4 experts, FP8 dense) by the author's own engine, so the quantized agreement bar applies. Gated short thinking on about 19% of questions with a deterministic think budget; serving settings were compared on Decision Index public samples (author-disclosed). Author public-run adoption/execution status and private lab setup are disclosed in configuration; evaluator-published does not mean every public run was independently executed. 32 benchmarks with a per-field right or wrong answer (ToolRet, BRIGHT and RouterBench excluded), each weighted equally; a 1-in-6 sample of requests; confidence is the probability placed on the chosen option. Rows an entrant trained on count as wrong here too. Contributing benchmarks 31. The board does not name a served checkpoint in this repository history. Decision Index 0.4 (release-v4), source generated 2026-10-20T14:13:56Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Checked / reported: 2026-11-02 · Samples: 34411 Snapshot: 2026-10-20T14:13:56Z · Revision: 133313cf8963f2b6bed960abb0152fd14cbb6731; c130faf65f82e81d3bdefb01e758c712dc5c5199 for the 2 November 2026 additions · Published observations; result reuse terms not specified Result source · Benchmark source
HF text calibration 0.4 · Knowledge · ECE	15.08 pp	Setup & source HF text calibration 0.4 · Knowledge · ECE Exact source engine span-02; {"kind": "full fine-tune", "technique": "autoregressive", "technique_name": "autoregressive", "served_params": 25805936206, "params_estimated": false, "params_from_name": false, "trained_bytes": null, "model_url": "https://huggingface.co/lihe07/span-02", "code_url": null, "base_model": "google/gemma-4-26B-A4B-it", "base_url": "https://huggingface.co/google/gemma-4-26B-A4B-it", "weights_repo": null, "served_checkpoint": null}. New in the 2026-10-14 PR batch (PR #171). Submitted public 0.3 run adopted (kit rescore 63.62, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving (NVFP4 experts, FP8 dense); decision agreement with the submitted rows 100.00% on 831 rows (bar 96%). Served quantized (NVFP4 experts, FP8 dense) by the author's own engine, so the quantized agreement bar applies. Gated short thinking on about 19% of questions with a deterministic think budget; serving settings were compared on Decision Index public samples (author-disclosed). Decision Index 0.4.1 (release-v4), source generated 2026-10-29T14:04:16Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Source notes: New in the 2026-10-14 PR batch (PR #171). Submitted public 0.3 run adopted (kit rescore 63.62, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving (NVFP4 experts, FP8 dense); decision agreement with the submitted rows 100.00% on 831 rows (bar 96%). Served quantized (NVFP4 experts, FP8 dense) by the author's own engine, so the quantized agreement bar applies. Gated short thinking on about 19% of questions with a deterministic think budget; serving settings were compared on Decision Index public samples (author-disclosed). Author public-run adoption/execution status and private lab setup are disclosed in configuration; evaluator-published does not mean every public run was independently executed. Area-specific item counts unpublished; do not use whole-sample count. The board does not name a served checkpoint in this repository history. Decision Index 0.4 (release-v4), source generated 2026-10-20T14:13:56Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Checked / reported: 2026-11-02 · Samples: Not published Snapshot: 2026-10-20T14:13:56Z · Revision: 133313cf8963f2b6bed960abb0152fd14cbb6731; c130faf65f82e81d3bdefb01e758c712dc5c5199 for the 2 November 2026 additions · Published observations; result reuse terms not specified Result source · Benchmark source
HF text calibration 0.4 · Knowledge · Sample accuracy	67.44%	Setup & source HF text calibration 0.4 · Knowledge · Sample accuracy Exact source engine span-02; {"kind": "full fine-tune", "technique": "autoregressive", "technique_name": "autoregressive", "served_params": 25805936206, "params_estimated": false, "params_from_name": false, "trained_bytes": null, "model_url": "https://huggingface.co/lihe07/span-02", "code_url": null, "base_model": "google/gemma-4-26B-A4B-it", "base_url": "https://huggingface.co/google/gemma-4-26B-A4B-it", "weights_repo": null, "served_checkpoint": null}. New in the 2026-10-14 PR batch (PR #171). Submitted public 0.3 run adopted (kit rescore 63.62, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving (NVFP4 experts, FP8 dense); decision agreement with the submitted rows 100.00% on 831 rows (bar 96%). Served quantized (NVFP4 experts, FP8 dense) by the author's own engine, so the quantized agreement bar applies. Gated short thinking on about 19% of questions with a deterministic think budget; serving settings were compared on Decision Index public samples (author-disclosed). Decision Index 0.4.1 (release-v4), source generated 2026-10-29T14:04:16Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Source notes: New in the 2026-10-14 PR batch (PR #171). Submitted public 0.3 run adopted (kit rescore 63.62, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving (NVFP4 experts, FP8 dense); decision agreement with the submitted rows 100.00% on 831 rows (bar 96%). Served quantized (NVFP4 experts, FP8 dense) by the author's own engine, so the quantized agreement bar applies. Gated short thinking on about 19% of questions with a deterministic think budget; serving settings were compared on Decision Index public samples (author-disclosed). Author public-run adoption/execution status and private lab setup are disclosed in configuration; evaluator-published does not mean every public run was independently executed. Area-specific item counts unpublished; do not use whole-sample count. The board does not name a served checkpoint in this repository history. Decision Index 0.4 (release-v4), source generated 2026-10-20T14:13:56Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Checked / reported: 2026-11-02 · Samples: Not published Snapshot: 2026-10-20T14:13:56Z · Revision: 133313cf8963f2b6bed960abb0152fd14cbb6731; c130faf65f82e81d3bdefb01e758c712dc5c5199 for the 2 November 2026 additions · Published observations; result reuse terms not specified Result source · Benchmark source
HF text calibration 0.4 · Knowledge · Mean chosen confidence	82.52%	Setup & source HF text calibration 0.4 · Knowledge · Mean chosen confidence Exact source engine span-02; {"kind": "full fine-tune", "technique": "autoregressive", "technique_name": "autoregressive", "served_params": 25805936206, "params_estimated": false, "params_from_name": false, "trained_bytes": null, "model_url": "https://huggingface.co/lihe07/span-02", "code_url": null, "base_model": "google/gemma-4-26B-A4B-it", "base_url": "https://huggingface.co/google/gemma-4-26B-A4B-it", "weights_repo": null, "served_checkpoint": null}. New in the 2026-10-14 PR batch (PR #171). Submitted public 0.3 run adopted (kit rescore 63.62, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving (NVFP4 experts, FP8 dense); decision agreement with the submitted rows 100.00% on 831 rows (bar 96%). Served quantized (NVFP4 experts, FP8 dense) by the author's own engine, so the quantized agreement bar applies. Gated short thinking on about 19% of questions with a deterministic think budget; serving settings were compared on Decision Index public samples (author-disclosed). Decision Index 0.4.1 (release-v4), source generated 2026-10-29T14:04:16Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Source notes: New in the 2026-10-14 PR batch (PR #171). Submitted public 0.3 run adopted (kit rescore 63.62, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving (NVFP4 experts, FP8 dense); decision agreement with the submitted rows 100.00% on 831 rows (bar 96%). Served quantized (NVFP4 experts, FP8 dense) by the author's own engine, so the quantized agreement bar applies. Gated short thinking on about 19% of questions with a deterministic think budget; serving settings were compared on Decision Index public samples (author-disclosed). Author public-run adoption/execution status and private lab setup are disclosed in configuration; evaluator-published does not mean every public run was independently executed. Area-specific item counts unpublished; do not use whole-sample count. The board does not name a served checkpoint in this repository history. Decision Index 0.4 (release-v4), source generated 2026-10-20T14:13:56Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Checked / reported: 2026-11-02 · Samples: Not published Snapshot: 2026-10-20T14:13:56Z · Revision: 133313cf8963f2b6bed960abb0152fd14cbb6731; c130faf65f82e81d3bdefb01e758c712dc5c5199 for the 2 November 2026 additions · Published observations; result reuse terms not specified Result source · Benchmark source
HF text calibration 0.4 · Language · ECE	5.11 pp	Setup & source HF text calibration 0.4 · Language · ECE Exact source engine span-02; {"kind": "full fine-tune", "technique": "autoregressive", "technique_name": "autoregressive", "served_params": 25805936206, "params_estimated": false, "params_from_name": false, "trained_bytes": null, "model_url": "https://huggingface.co/lihe07/span-02", "code_url": null, "base_model": "google/gemma-4-26B-A4B-it", "base_url": "https://huggingface.co/google/gemma-4-26B-A4B-it", "weights_repo": null, "served_checkpoint": null}. New in the 2026-10-14 PR batch (PR #171). Submitted public 0.3 run adopted (kit rescore 63.62, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving (NVFP4 experts, FP8 dense); decision agreement with the submitted rows 100.00% on 831 rows (bar 96%). Served quantized (NVFP4 experts, FP8 dense) by the author's own engine, so the quantized agreement bar applies. Gated short thinking on about 19% of questions with a deterministic think budget; serving settings were compared on Decision Index public samples (author-disclosed). Decision Index 0.4.1 (release-v4), source generated 2026-10-29T14:04:16Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Source notes: New in the 2026-10-14 PR batch (PR #171). Submitted public 0.3 run adopted (kit rescore 63.62, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving (NVFP4 experts, FP8 dense); decision agreement with the submitted rows 100.00% on 831 rows (bar 96%). Served quantized (NVFP4 experts, FP8 dense) by the author's own engine, so the quantized agreement bar applies. Gated short thinking on about 19% of questions with a deterministic think budget; serving settings were compared on Decision Index public samples (author-disclosed). Author public-run adoption/execution status and private lab setup are disclosed in configuration; evaluator-published does not mean every public run was independently executed. Area-specific item counts unpublished; do not use whole-sample count. The board does not name a served checkpoint in this repository history. Decision Index 0.4 (release-v4), source generated 2026-10-20T14:13:56Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Checked / reported: 2026-11-02 · Samples: Not published Snapshot: 2026-10-20T14:13:56Z · Revision: 133313cf8963f2b6bed960abb0152fd14cbb6731; c130faf65f82e81d3bdefb01e758c712dc5c5199 for the 2 November 2026 additions · Published observations; result reuse terms not specified Result source · Benchmark source
HF text calibration 0.4 · Language · Sample accuracy	86.35%	Setup & source HF text calibration 0.4 · Language · Sample accuracy Exact source engine span-02; {"kind": "full fine-tune", "technique": "autoregressive", "technique_name": "autoregressive", "served_params": 25805936206, "params_estimated": false, "params_from_name": false, "trained_bytes": null, "model_url": "https://huggingface.co/lihe07/span-02", "code_url": null, "base_model": "google/gemma-4-26B-A4B-it", "base_url": "https://huggingface.co/google/gemma-4-26B-A4B-it", "weights_repo": null, "served_checkpoint": null}. New in the 2026-10-14 PR batch (PR #171). Submitted public 0.3 run adopted (kit rescore 63.62, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving (NVFP4 experts, FP8 dense); decision agreement with the submitted rows 100.00% on 831 rows (bar 96%). Served quantized (NVFP4 experts, FP8 dense) by the author's own engine, so the quantized agreement bar applies. Gated short thinking on about 19% of questions with a deterministic think budget; serving settings were compared on Decision Index public samples (author-disclosed). Decision Index 0.4.1 (release-v4), source generated 2026-10-29T14:04:16Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Source notes: New in the 2026-10-14 PR batch (PR #171). Submitted public 0.3 run adopted (kit rescore 63.62, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving (NVFP4 experts, FP8 dense); decision agreement with the submitted rows 100.00% on 831 rows (bar 96%). Served quantized (NVFP4 experts, FP8 dense) by the author's own engine, so the quantized agreement bar applies. Gated short thinking on about 19% of questions with a deterministic think budget; serving settings were compared on Decision Index public samples (author-disclosed). Author public-run adoption/execution status and private lab setup are disclosed in configuration; evaluator-published does not mean every public run was independently executed. Area-specific item counts unpublished; do not use whole-sample count. The board does not name a served checkpoint in this repository history. Decision Index 0.4 (release-v4), source generated 2026-10-20T14:13:56Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Checked / reported: 2026-11-02 · Samples: Not published Snapshot: 2026-10-20T14:13:56Z · Revision: 133313cf8963f2b6bed960abb0152fd14cbb6731; c130faf65f82e81d3bdefb01e758c712dc5c5199 for the 2 November 2026 additions · Published observations; result reuse terms not specified Result source · Benchmark source
HF text calibration 0.4 · Language · Mean chosen confidence	91.46%	Setup & source HF text calibration 0.4 · Language · Mean chosen confidence Exact source engine span-02; {"kind": "full fine-tune", "technique": "autoregressive", "technique_name": "autoregressive", "served_params": 25805936206, "params_estimated": false, "params_from_name": false, "trained_bytes": null, "model_url": "https://huggingface.co/lihe07/span-02", "code_url": null, "base_model": "google/gemma-4-26B-A4B-it", "base_url": "https://huggingface.co/google/gemma-4-26B-A4B-it", "weights_repo": null, "served_checkpoint": null}. New in the 2026-10-14 PR batch (PR #171). Submitted public 0.3 run adopted (kit rescore 63.62, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving (NVFP4 experts, FP8 dense); decision agreement with the submitted rows 100.00% on 831 rows (bar 96%). Served quantized (NVFP4 experts, FP8 dense) by the author's own engine, so the quantized agreement bar applies. Gated short thinking on about 19% of questions with a deterministic think budget; serving settings were compared on Decision Index public samples (author-disclosed). Decision Index 0.4.1 (release-v4), source generated 2026-10-29T14:04:16Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Source notes: New in the 2026-10-14 PR batch (PR #171). Submitted public 0.3 run adopted (kit rescore 63.62, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving (NVFP4 experts, FP8 dense); decision agreement with the submitted rows 100.00% on 831 rows (bar 96%). Served quantized (NVFP4 experts, FP8 dense) by the author's own engine, so the quantized agreement bar applies. Gated short thinking on about 19% of questions with a deterministic think budget; serving settings were compared on Decision Index public samples (author-disclosed). Author public-run adoption/execution status and private lab setup are disclosed in configuration; evaluator-published does not mean every public run was independently executed. Area-specific item counts unpublished; do not use whole-sample count. The board does not name a served checkpoint in this repository history. Decision Index 0.4 (release-v4), source generated 2026-10-20T14:13:56Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Checked / reported: 2026-11-02 · Samples: Not published Snapshot: 2026-10-20T14:13:56Z · Revision: 133313cf8963f2b6bed960abb0152fd14cbb6731; c130faf65f82e81d3bdefb01e758c712dc5c5199 for the 2 November 2026 additions · Published observations; result reuse terms not specified Result source · Benchmark source
HF text calibration 0.4 · Retrieval · ECE	10.91 pp	Setup & source HF text calibration 0.4 · Retrieval · ECE Exact source engine span-02; {"kind": "full fine-tune", "technique": "autoregressive", "technique_name": "autoregressive", "served_params": 25805936206, "params_estimated": false, "params_from_name": false, "trained_bytes": null, "model_url": "https://huggingface.co/lihe07/span-02", "code_url": null, "base_model": "google/gemma-4-26B-A4B-it", "base_url": "https://huggingface.co/google/gemma-4-26B-A4B-it", "weights_repo": null, "served_checkpoint": null}. New in the 2026-10-14 PR batch (PR #171). Submitted public 0.3 run adopted (kit rescore 63.62, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving (NVFP4 experts, FP8 dense); decision agreement with the submitted rows 100.00% on 831 rows (bar 96%). Served quantized (NVFP4 experts, FP8 dense) by the author's own engine, so the quantized agreement bar applies. Gated short thinking on about 19% of questions with a deterministic think budget; serving settings were compared on Decision Index public samples (author-disclosed). Decision Index 0.4.1 (release-v4), source generated 2026-10-29T14:04:16Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Source notes: New in the 2026-10-14 PR batch (PR #171). Submitted public 0.3 run adopted (kit rescore 63.62, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving (NVFP4 experts, FP8 dense); decision agreement with the submitted rows 100.00% on 831 rows (bar 96%). Served quantized (NVFP4 experts, FP8 dense) by the author's own engine, so the quantized agreement bar applies. Gated short thinking on about 19% of questions with a deterministic think budget; serving settings were compared on Decision Index public samples (author-disclosed). Author public-run adoption/execution status and private lab setup are disclosed in configuration; evaluator-published does not mean every public run was independently executed. Area-specific item counts unpublished; do not use whole-sample count. The board does not name a served checkpoint in this repository history. Decision Index 0.4 (release-v4), source generated 2026-10-20T14:13:56Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Checked / reported: 2026-11-02 · Samples: Not published Snapshot: 2026-10-20T14:13:56Z · Revision: 133313cf8963f2b6bed960abb0152fd14cbb6731; c130faf65f82e81d3bdefb01e758c712dc5c5199 for the 2 November 2026 additions · Published observations; result reuse terms not specified Result source · Benchmark source
HF text calibration 0.4 · Retrieval · Sample accuracy	76.22%	Setup & source HF text calibration 0.4 · Retrieval · Sample accuracy Exact source engine span-02; {"kind": "full fine-tune", "technique": "autoregressive", "technique_name": "autoregressive", "served_params": 25805936206, "params_estimated": false, "params_from_name": false, "trained_bytes": null, "model_url": "https://huggingface.co/lihe07/span-02", "code_url": null, "base_model": "google/gemma-4-26B-A4B-it", "base_url": "https://huggingface.co/google/gemma-4-26B-A4B-it", "weights_repo": null, "served_checkpoint": null}. New in the 2026-10-14 PR batch (PR #171). Submitted public 0.3 run adopted (kit rescore 63.62, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving (NVFP4 experts, FP8 dense); decision agreement with the submitted rows 100.00% on 831 rows (bar 96%). Served quantized (NVFP4 experts, FP8 dense) by the author's own engine, so the quantized agreement bar applies. Gated short thinking on about 19% of questions with a deterministic think budget; serving settings were compared on Decision Index public samples (author-disclosed). Decision Index 0.4.1 (release-v4), source generated 2026-10-29T14:04:16Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Source notes: New in the 2026-10-14 PR batch (PR #171). Submitted public 0.3 run adopted (kit rescore 63.62, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving (NVFP4 experts, FP8 dense); decision agreement with the submitted rows 100.00% on 831 rows (bar 96%). Served quantized (NVFP4 experts, FP8 dense) by the author's own engine, so the quantized agreement bar applies. Gated short thinking on about 19% of questions with a deterministic think budget; serving settings were compared on Decision Index public samples (author-disclosed). Author public-run adoption/execution status and private lab setup are disclosed in configuration; evaluator-published does not mean every public run was independently executed. Area-specific item counts unpublished; do not use whole-sample count. The board does not name a served checkpoint in this repository history. Decision Index 0.4 (release-v4), source generated 2026-10-20T14:13:56Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Checked / reported: 2026-11-02 · Samples: Not published Snapshot: 2026-10-20T14:13:56Z · Revision: 133313cf8963f2b6bed960abb0152fd14cbb6731; c130faf65f82e81d3bdefb01e758c712dc5c5199 for the 2 November 2026 additions · Published observations; result reuse terms not specified Result source · Benchmark source
HF text calibration 0.4 · Retrieval · Mean chosen confidence	87.13%	Setup & source HF text calibration 0.4 · Retrieval · Mean chosen confidence Exact source engine span-02; {"kind": "full fine-tune", "technique": "autoregressive", "technique_name": "autoregressive", "served_params": 25805936206, "params_estimated": false, "params_from_name": false, "trained_bytes": null, "model_url": "https://huggingface.co/lihe07/span-02", "code_url": null, "base_model": "google/gemma-4-26B-A4B-it", "base_url": "https://huggingface.co/google/gemma-4-26B-A4B-it", "weights_repo": null, "served_checkpoint": null}. New in the 2026-10-14 PR batch (PR #171). Submitted public 0.3 run adopted (kit rescore 63.62, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving (NVFP4 experts, FP8 dense); decision agreement with the submitted rows 100.00% on 831 rows (bar 96%). Served quantized (NVFP4 experts, FP8 dense) by the author's own engine, so the quantized agreement bar applies. Gated short thinking on about 19% of questions with a deterministic think budget; serving settings were compared on Decision Index public samples (author-disclosed). Decision Index 0.4.1 (release-v4), source generated 2026-10-29T14:04:16Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Source notes: New in the 2026-10-14 PR batch (PR #171). Submitted public 0.3 run adopted (kit rescore 63.62, equal to the claim). Private tests run by the lab on one RTX PRO 6000 with the author's serving (NVFP4 experts, FP8 dense); decision agreement with the submitted rows 100.00% on 831 rows (bar 96%). Served quantized (NVFP4 experts, FP8 dense) by the author's own engine, so the quantized agreement bar applies. Gated short thinking on about 19% of questions with a deterministic think budget; serving settings were compared on Decision Index public samples (author-disclosed). Author public-run adoption/execution status and private lab setup are disclosed in configuration; evaluator-published does not mean every public run was independently executed. Area-specific item counts unpublished; do not use whole-sample count. The board does not name a served checkpoint in this repository history. Decision Index 0.4 (release-v4), source generated 2026-10-20T14:13:56Z. Full = 60% equated private same-skill + 40% equated private new-domain tasks; public contributes zero. Both private components are mapped onto the public scale using untuned stock models. Public weighting/panel changed (GSM8K rebuilt, ForecastBench retired, WinoGrande area reassigned); these are new benchmark definitions, not proof of a model upgrade. Public 37-benchmark index and 42 published native benchmarks remain separate. Preserve source rounding, coverage, exclusions and unknown execution dates; no new uncertainty inferred. Checked / reported: 2026-11-02 · Samples: Not published Snapshot: 2026-10-20T14:13:56Z · Revision: 133313cf8963f2b6bed960abb0152fd14cbb6731; c130faf65f82e81d3bdefb01e758c712dc5c5199 for the 2 November 2026 additions · Published observations; result reuse terms not specified Result source · Benchmark source